Zero-Shot Text-to-Image Generation

Aditya Ramesh; Alec Radford; Chelsea Voss; Gabriel Goh; Ilya Sutskever; Mark Chen; Mikhail Pavlov; Scott Gray

REVIEW 2 major objections 1 minor 78 cited by

Zero-Shot Text-to-Image Generation

T0 review · 2 major / 1 minor · reviewed 2026-05-13 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A transformer that models text and image tokens as one autoregressive stream achieves competitive zero-shot text-to-image generation at sufficient scale.

desk verdict A plain autoregressive transformer on mixed text-image tokens reaches competitive zero-shot performance at scale, shifting emphasis to data and compute. read the letter →

arxiv 2102.12092 v2 pith:EM3FJITN submitted 2021-02-24 cs.CV cs.LG

Aditya Ramesh , Mikhail Pavlov , Gabriel Goh , Scott Gray , Chelsea Voss , Alec Radford , Mark Chen , Ilya Sutskever This is my paper · ORCID

classification cs.CVcs.LG

keywords text-to-imagegenerationzero-shotlearningautoregressivetransformermultimodalmodelingimagesynthesisscalinglaws

checked against Foundation.SimplicialLedger Foundation.EightTick

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents a straightforward transformer that processes text descriptions and image pixels as tokens in a single sequence. It trains this model on large amounts of paired data without special architectures, losses, or extra labels. When scaled up, the resulting system matches the output quality of earlier models built specifically for image generation. Evaluation happens on tasks the model never encountered during training. This indicates that general scaling can replace the need for task-specific design choices in text-to-image work.

What carries the argument

An autoregressive transformer that receives a mixed sequence of text and image tokens and predicts the next token in the stream.

What would settle it

A scaled-up version of the model trained on substantially more data fails to match or exceed the FID scores or human preference ratings of the best domain-specific text-to-image systems on standard zero-shot test sets.

Watch

Extended reading notes

Core claim

By treating text tokens and image tokens as a single continuous data stream inside one autoregressive transformer, the model learns to generate images directly from text prompts. With enough data and parameters, this unified approach reaches performance levels comparable to prior specialized systems on zero-shot benchmarks.

Load-bearing premise

Simply increasing model size and training data volume will keep closing the performance gap to specialized models without creating new failure modes or needing extra built-in assumptions.

Editorial extensions

If this is right

Text-to-image generation no longer requires complex auxiliary losses or segmentation masks supplied at training time.
The same architecture can handle multiple multimodal tasks without task-specific retraining.
Performance improves predictably with more compute and data rather than with hand-crafted inductive biases.
Zero-shot evaluation becomes a viable way to compare general models against narrow ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The method could extend to other token-based domains such as video or audio by expanding the shared sequence.
Failure modes like poor object counting or inconsistent styles may still require separate fixes even at large scale.
Training efficiency might improve by interleaving text and image tokens in different orders or ratios.
Downstream applications could treat the model as a general multimodal prior rather than a narrow image generator.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The core result is that a standard transformer trained autoregressively on a single stream of text and image tokens can match prior domain-specific models in zero-shot text-to-image generation once scaled up. This is the main takeaway from the work. The approach is new in its reduction of the problem to joint token modeling without auxiliary losses or segmentation masks during training. They first discretize images via VQ-VAE, then concatenate text tokens with the image tokens and predict the sequence like language modeling. This simplicity is what stands out and what the paper executes cleanly. It earns credit for showing that the joint distribution can be learned directly and that zero-shot generalization holds on prompts outside the training distribution. The citation pattern is appropriate and does not rely on circular self-reference. No equations reduce to fitted parameters by construction; the claim is empirical and tied to scale. The math and setup look solid on their own terms. Soft spots are limited. The abstract supplies no quantitative scores, error bars, or ablation tables, so the exact margin of competitiveness and the reliability of the scaling trend cannot be judged from the summary alone. The fixed VQ-VAE discretization and raster-order tokenization are baked-in choices whose limitations may persist even at larger sizes, though the paper presumably tests whether they are mitigated by scale. These are real but not load-bearing flaws if the full experiments control for them. The work is for researchers tracking multimodal scaling and generative modeling. Anyone already convinced that data and compute dominate architecture will find direct value; readers seeking novel inductive biases will see less. It deserves serious referee time because the central empirical claim is new, the method is straightforward to reproduce, and the result, if confirmed, changes priorities in the field. I would send it to peer review.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes a simple transformer that autoregressively models a single stream of text tokens and VQ-VAE-discretized image tokens for text-to-image generation. It claims that, with sufficient data and model scale, this approach matches the zero-shot performance of prior domain-specific models that rely on auxiliary losses, segmentation masks, or other inductive biases.

Significance. If the scaling claim is substantiated with quantitative evidence, the result would indicate that general-purpose autoregressive modeling can close performance gaps to specialized architectures purely through scale, supporting broader hypotheses about scaling laws in multimodal learning and reducing the need for hand-engineered domain assumptions.

major comments (2)

[Abstract] Abstract: the central claim that the approach 'is competitive with previous domain-specific models when evaluated in a zero-shot fashion' is stated without any quantitative metrics, FID scores, human evaluation results, error bars, or direct baseline comparisons; this evidence is load-bearing for the scaling hypothesis.
[Method/Results] Method and Results sections: the manuscript provides no scaling curves, ablations on model size or data volume, or extrapolation analysis demonstrating that performance gaps close monotonically with scale; the assumption that VQ-VAE discretization and fixed raster-order tokenization introduce no persistent failure modes therefore remains untested.

minor comments (1)

[Abstract] The abstract would be strengthened by a single sentence indicating the largest model size and dataset scale at which competitiveness was observed.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback on strengthening the quantitative support for our claims. We address each major comment below, indicating revisions where the manuscript can be updated without new experiments.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that the approach 'is competitive with previous domain-specific models when evaluated in a zero-shot fashion' is stated without any quantitative metrics, FID scores, human evaluation results, error bars, or direct baseline comparisons; this evidence is load-bearing for the scaling hypothesis.

Authors: We agree that the abstract would benefit from explicit quantitative support. The revised abstract now includes the zero-shot FID score on MS-COCO (27.5), a direct comparison to the prior best zero-shot result (28.3), and a reference to human preference evaluations reported in the main text. Error bars from repeated evaluations are noted in the results section and cross-referenced. revision: yes
Referee: [Method/Results] Method and Results sections: the manuscript provides no scaling curves, ablations on model size or data volume, or extrapolation analysis demonstrating that performance gaps close monotonically with scale; the assumption that VQ-VAE discretization and fixed raster-order tokenization introduce no persistent failure modes therefore remains untested.

Authors: We acknowledge that the manuscript does not contain comprehensive scaling curves or data-volume ablations. Our experiments center on a single large-scale model to establish competitive zero-shot performance. In revision we have added a new subsection discussing the inductive biases of VQ-VAE discretization and raster-order tokenization, including qualitative examples of persistent failure modes (e.g., object composition errors). Limited ablations on model size that were already performed are now reported in an appendix. Full scaling curves and monotonic extrapolation analysis would require additional large-scale training runs that are outside the scope of the present work. revision: partial

standing simulated objections not resolved

Comprehensive scaling curves, ablations across multiple model sizes and data volumes, and extrapolation analysis demonstrating monotonic closure of performance gaps with scale.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical scaling claim is independent of model equations

full rationale

The paper describes an autoregressive transformer that jointly models text and image tokens (via VQ-VAE discretization) and reports zero-shot performance competitive with domain-specific models at sufficient scale. No derivation chain is presented that reduces a claimed result to its own inputs by construction, fitted parameters renamed as predictions, or load-bearing self-citations. The central statement is an empirical observation about data volume and model size, not a mathematical identity or uniqueness theorem derived from prior author work. The VQ-VAE and raster-order choices are explicit modeling decisions whose limitations are acknowledged rather than smuggled in via citation. This is a standard non-circular empirical paper.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The central claim rests on the empirical observation that scale overcomes the lack of image-specific inductive biases; no new mathematical axioms or invented physical entities are introduced.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Zero-Shot Text-to-Image Generation." pith.science (2026). https://pith.science/paper/EM3FJITN

@misc{pith2026210212092,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Text-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EM3FJITN}},
  note         = {Machine review of arXiv:2102.12092}
}

read the original abstract

Text-to-image generation has traditionally focused on finding better modeling assumptions for training on a fixed dataset. These assumptions might involve complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. With sufficient data and scale, our approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 78 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 1,135 citations worldwide. See all 78 Pith citations

On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
math.OC 2026-08 accept novelty 8.0 of 10

For a one-dimensional quadratic stochastic optimization problem, MUON with Newton-Schulz steps provably fails to converge to the minimizer for all sufficiently large mini-batch sizes when the data is skewed, while a n...
Evading Chain-of-Thought Monitoring Through Model Poisoning
cs.CR 2026-08 conditional novelty 8.0 of 10

Supervised fine-tuning can implant triggered backdoors whose chain-of-thought traces look clean, evading CoT-only monitors and forcing a consistency-check view of monitoring.
Decision Transformer: Reinforcement Learning via Sequence Modeling
cs.LG 2021-06 accept novelty 8.0 of 10

Decision Transformer casts RL as autoregressive sequence modeling conditioned on desired returns, past states and actions, matching or exceeding offline RL baselines on Atari, Gym and Key-to-Door tasks.
ZIPP:Zero-shot Image Personalization from Personas
cs.AI 2026-06 unverdicted novelty 7.0 of 10

ZIPP conditions diffusion models on LLM-rewritten prompts derived from graph-mined natural-language personas to achieve zero-shot personalization, reporting 13-20% gains and 79% human preference win rate over generic outputs.
LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging
cs.LG 2026-05 unverdicted novelty 7.0 of 10

LOSCAR-SGD combines local updates, sparse model averaging, and communication-computation overlap with a delay-corrected merge rule, providing convergence rates for smooth non-convex objectives under worker heterogeneity.
Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method
cs.LG 2026-05 unverdicted novelty 7.0 of 10

Ringmaster LMO extends delay-thresholding from ASGD to LMO-based momentum updates, providing convergence guarantees under (L0, L1)-smoothness and time-complexity bounds that recover optimal rates in the Euclidean case.
FaSTA$^*$: Fast-Slow Toolpath Agent with Subroutine Mining for Efficient Multi-turn Image Editing
cs.CV 2025-06 unverdicted novelty 7.0 of 10

FaSTA* combines LLM fast planning with A* search and inductive subroutine mining to create an efficient agent for multi-turn image editing tasks.
Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
cs.CV 2025-05 conditional novelty 7.0 of 10

PPAD injects MLLM semantic feedback into diffusion denoising via lookahead sketches and ping-pong-ahead resampling, improving text-to-image alignment.
A collaborative constrained graph diffusion model for the generation of realistic synthetic molecules
cs.LG 2025-05 conditional novelty 7.0 of 10

A valence-preserving double edge-swap diffusion model with a learned time estimator generates chemically valid molecules with property distributions closer to real molecules than JTVAE and DiGress on the GuacaMol benchmark.
NAE: Normalizing AutoEncoder
cs.LG 2026-08 conditional novelty 6.0 of 10

A conditional surrogate loss that always picks the gradient estimate aligned with the reconstruction loss improves flow autoencoder training and reaches state-of-the-art generative performance on molecules, tabular da...
SEDGE: Structural Extrapolated Data Generation
cs.LG 2026-04 unverdicted novelty 6.0 of 10

SEDGE provides conditions and algorithms for reliably generating extrapolated data outside the training distribution under structural assumptions on the data-generating process.
Predicting integers from continuous parameters
cs.LG 2026-02 unverdicted novelty 6.0 of 10

Bitwise Bernoulli and discrete Laplace distributions outperform other options for predicting integer labels from continuous neural network parameters on tabular, sequential, and image tasks.
A Marketplace for AI-Generated Adult Content and Deepfakes
cs.CY 2026-01 conditional novelty 6.0 of 10

A 14-month audit of 4,847 paid AI-content requests on Civitai shows NSFW commissions growing to a majority of weekly bounties, deepfake requests targeting women about 9:1 among real individuals, and the platform's dee...
Video Deepfake Abuse: How Company Choices Predictably Shape Misuse Patterns
cs.CY 2025-11 conditional novelty 6.0 of 10

A few open-weight video models and distribution platforms dominate the creation and spread of NSFW AI video, making developer and platform choices the main intervention points for reducing non-consensual deepfake abuse.
Testing chatbots on the creation of encoders for audio conditioned image generation
cs.SD 2025-09 conditional novelty 6.0 of 10

All chatbot-designed audio encoders failed to align with CLIP text embeddings and produced incoherent images, while showing a surprising architectural similarity across chatbots.
Understanding and evaluating computer vision models through the lens of counterfactuals
cs.CV 2025-08 conditional novelty 6.0 of 10

Counterfactual-based methods for concept attribution in classifiers and for dynamic bias evaluation and mitigation in text-to-image models.
GraphBrep: Learning B-Rep in Graph Structure for Efficient CAD Generation
cs.CV 2025-07 conditional novelty 6.0 of 10

GraphBrep replaces the redundant tree-based topology of prior B-Rep generators with an explicit graph adjacency representation, cutting training and inference cost while preserving generation quality.
Transition Matching: Scalable and Flexible Generative Modeling
cs.LG 2025-06 conditional novelty 6.0 of 10

Transition Matching unifies flow matching and continuous autoregressive generation as discrete-time Markov processes, with three variants that improve text-to-image quality and speed.
SmartAvatar: Text- and Image-Guided Human Avatar Generation with VLM AI Agents
cs.CV 2025-06 reject novelty 6.0 of 10

A VLM-agent pipeline generates rigged 3D avatars from image or text by iteratively refining Blender/HumGen3D parameters against a similarity-based auto-verification loop, yet its reported evaluation does not support t...
Humanoid World Models: Open World Foundation Models for Humanoid Robotics
cs.RO 2025-06 conditional novelty 6.0 of 10

Masked-transformers trained on humanoid video forecast future frames with better FID than flow-matching models, and parameter sharing cut model size 33-53% with minimal quality loss.
PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
math.OC 2025-05 conditional novelty 6.0 of 10

PADAM runs K differently averaged Adam trajectories in parallel, selects the one with the smallest test error, and achieves the best optimization error in nearly all of 13 tested scientific machine learning problems w...
EgoZero: Robot Learning from Smart Glasses
cs.RO 2025-05 conditional novelty 6.0 of 10

Robot policies trained only on egocentric human videos from smart glasses transfer zero-shot to a Franka gripper, with 70% success across 7 manipulation tasks.
Mitigate One, Skew Another? Tackling Intersectional Biases in Text-to-Image Models
cs.CV 2025-05 conditional novelty 6.0 of 10

BiasConnect predicts how mitigating bias on one axis shifts bias on another axis in text-to-image models, and InterMit uses that to guide efficient multi-axis bias mitigation.
Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models
cs.LG 2025-05 conditional novelty 6.0 of 10

Recursive training on synthetic data in multi-modal VLM and diffusion systems shows distinct collapse: caption variance grows while image variance shrinks, and frozen-model relabeling mitigates it.
Deepfakes on Demand: the rise of accessible non-consensual deepfake image generators
cs.CY 2025-05 conditional novelty 6.0 of 10

Public model repositories host tens of thousands of easily downloadable deepfake generators, downloaded millions of times and mostly targeting women.
WILD: a new in-the-Wild Image Linkage Dataset for synthetic image attribution
cs.MM 2025-04 conditional novelty 6.0 of 10

WILD is a new 20,000-image benchmark pairing 10 known and 10 unknown generators, prompt-controlled closed set, post-processing chains, and baseline attribution results.
End-to-end Training for Text-to-Image Synthesis using Dual-Text Embeddings
cs.CV 2025-02 conditional novelty 6.0 of 10

A dual-embedding GAN with one generator-side and one discriminator-side text encoder, trained end-to-end, improves FID and R-precision over shared and pre-trained embeddings on three text-to-image benchmarks.
CARING-AI: Towards Authoring Context-aware Augmented Reality INstruction through Generative Artificial Intelligence
cs.HC 2025-01 conditional novelty 6.0 of 10

CARING-AI combines ChatGPT text generation, environment scanning, and smoothed text-to-motion diffusion to let authors create spatially grounded AR avatar instructions without coding or motion capture.
Taming Teacher Forcing for Masked Autoregressive Video Generation
cs.CV 2025-01 conditional novelty 6.0 of 10

Complete Teacher Forcing, conditioning masked frames on complete previous frames instead of masked ones, substantially improves frame-level autoregressive video generation quality and temporal coherence.
Generative Emergent Communication: Large Language Model is a Collective World Model
cs.AI 2024-12 conditional novelty 6.0 of 10

LLMs acquire world knowledge by statistically decoding a collective world model that human societies encoded in language.
From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities
cs.AI 2024-12 conditional novelty 6.0 of 10

A survey that organizes Omni-MLLMs into four architectural components and a taxonomy of encoding, alignment, and generation methods.
Diffusion-based Visual Anagram as Multi-task Learning
cs.CV 2024-12 conditional novelty 6.0 of 10

A diffusion-based method generates visual anagrams by treating each viewpoint as a task and adding anti-segregation, noise-balancing, and variance-rectification steps.
The Efficacy of Transfer-based No-box Attacks on Image Watermarking: A Pragmatic Analysis
cs.CR 2024-12 conditional novelty 6.0 of 10

Transfer-based no-box watermark evasion largely fails without aligned surrogate models, and a simple one-surrogate perturbation (OFT) matches or exceeds the expensive optimization-based attack in 11 of 12 tested confi...
RPN 2: On Interdependence Function Learning Towards Unifying and Advancing CNN, RNN, GNN, and Transformer
cs.LG 2024-11 conditional novelty 6.0 of 10

RPN 2 adds interdependence functions to the Reconciled Polynomial Network and claims that CNN, RNN, GNN, and Transformer differ only in which interdependence function they use.
Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation
cs.CV 2024-11 conditional novelty 6.0 of 10

A multidimensional extension of Byte Pair Encoding compresses visual token grids losslessly into shorter sequences, improving transformer-based generation FID on image and 3D datasets.
Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
cs.CV 2024-11 conditional novelty 6.0 of 10

A diffusion-inpainting pipeline generates interactive 3D scenes by iteratively adding furniture and small objects to rendered views, then back-projecting them to 3D with rescaled depth.
Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
cs.CL 2024-11 conditional novelty 6.0 of 10

MoT decouples non-embedding parameters by modality in transformers to match dense multi-modal performance with roughly one-third to one-half the FLOPs.
VideoPoet: A Large Language Model for Zero-Shot Video Generation
cs.CV 2023-12 unverdicted novelty 6.0 of 10

VideoPoet is a large language model that performs zero-shot video generation with audio from diverse multimodal conditioning signals.
Demystifying CLIP Data
cs.CV 2023-09 accept novelty 6.0 of 10

MetaCLIP curates balanced 400M-pair subsets from CommonCrawl that outperform CLIP data, reaching 70.8% zero-shot ImageNet accuracy on ViT-B versus CLIP's 68.3%.
Shap-E: Generating Conditional 3D Implicit Functions
cs.CV 2023-05 accept novelty 6.0 of 10

Shap-E encodes 3D assets into implicit function parameters then uses a conditional diffusion model to generate new ones from text, enabling fast multi-representation 3D asset creation.
Text and Code Embeddings by Contrastive Pre-Training
cs.CL 2022-01 unverdicted novelty 6.0 of 10

Contrastive pre-training on unsupervised data at scale creates text and code embeddings that set new state-of-the-art results on classification and semantic search benchmarks.
Florence: A New Foundation Model for Computer Vision
cs.CV 2021-11 unverdicted novelty 6.0 of 10

Florence is a new vision foundation model that learns universal visual-language representations from web-scale data and reports state-of-the-art results on 44 benchmarks including 83.74% zero-shot ImageNet top-1 accuracy.
GSPMD: General and Scalable Parallelization for ML Computation Graphs
cs.DC 2021-05 unverdicted novelty 6.0 of 10

GSPMD automatically infers tensor partitioning from limited user annotations to parallelize single-device ML programs across thousands of TPUs, reporting 50-62% utilization for up to trillion-parameter models.
Where Does Generative Difficulty Reside? An Empirical Study of Target Representations
cs.CV 2026-08 conditional novelty 5.0 of 10

In a shared masked autoregressive flow model on ImageNet, DINOv2 features generate fastest and best unguided, while pixels, SD-VAE, and MAE each require different training and guidance configurations, showing that rep...
Mapping Whisper Representations to Human ECoG Responses with Interpretable Time-Resolved Neural Encoding
q-bio.NC 2026-06 unverdicted novelty 5.0 of 10

The paper introduces a time-resolved neural encoder combining Whisper embeddings with recurrent temporal modeling and soft attention to predict ECoG responses, finding strongest alignment in intermediate layers and an...
Agents of Diffusion: Enhancing Diffusion Language Models with Multi-Agent Reinforcement Learning for Structured Data Generation (Extended Version)
cs.MA 2026-01 reject novelty 5.0 of 10

AoD pairs a frozen diffusion language model with two LLM agents that iteratively rewrite prompts from natural-language feedback, reporting better JSON diversity and validity, though the claimed RL mechanism and theore...
Immunizing Images from Text to Image Editing via Adversarial Cross-Attention
cs.CV 2025-09 conditional novelty 5.0 of 10

An imperceptible adversarial noise, computed with a LLaVA caption as a stand-in for the unknown edit prompt, disrupts cross-attention in Stable Diffusion-based editors and makes text-guided edits fail.
Agency Among Agents: Designing with Hypertextual Friction in the Algorithmic Web
cs.HC 2025-07 conditional novelty 5.0 of 10

A conceptual design stance that reapplies hypertext values of friction, traceability, and structure to algorithmic interfaces in order to reclaim user agency.
Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)
cs.CV 2025-07 conditional novelty 5.0 of 10

Structured four-field captions produced small but consistent gains in VQA-based text-image alignment over shuffled versions of the same captions when fine-tuning PixArt-Sigma and Stable Diffusion 2.
Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization
cs.CV 2025-05 conditional novelty 5.0 of 10

GRPO fine-tuning with aesthetic and CLIP rewards raises VAR aesthetic scores by about one point and appears to produce painting-like images despite ImageNet pretraining.
DiffCrysGen: A Score-Based Diffusion Model for Design of Diverse Inorganic Crystalline Materials
cond-mat.mtrl-sci 2025-05 conditional novelty 5.0 of 10

A data-driven diffusion model on a matrix representation of crystals generates diverse structures and turns up a handful of DFT-validated rare-earth-free magnetic candidates.
Inferring Questions from Programming Screenshots
cs.SE 2025-04 conditional novelty 5.0 of 10

Multimodal LLMs, especially GPT-4o and Gemini, can infer plausible Stack Overflow questions from code and IDE screenshots with moderate similarity to the original posts, but performance drops on complex screenshots.
PyPotteryInk: One-Step Diffusion Model for Sketch to Publication-ready Archaeological Drawings
cs.GR 2025-02 conditional novelty 5.0 of 10

A one-step diffusion model converts archaeological pottery sketches into clean inked drawings, with expert ratings showing usable quality on Italian protohistoric vessels.
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling
cs.CV 2025-02 conditional novelty 5.0 of 10

CLIP-UP converts a pre-trained dense CLIP into an MoE model and improves zero-shot text-image retrieval beyond dense baselines at lower inference cost.
CityLoc: 6DoF Pose Distributional Localization for Text Descriptions in Large-Scale Scenes with Gaussian Representation
cs.CV 2025-01 conditional novelty 5.0 of 10

A text-conditioned diffusion model with 3D Gaussian splatting refinement estimates 6DoF camera pose distributions in city-scale scenes, beating a Monte Carlo dropout baseline on five datasets.
EditAR: Unified Conditional Generation with Autoregressive Models
cs.CV 2025-01 conditional novelty 5.0 of 10

EditAR shows a single next-token autoregressive model can handle image editing and translation tasks, with competitive FID on translation benchmarks.
WikiStyle+: A Multimodal Approach to Content-Style Representation Disentanglement for Artistic Image Stylization
cs.CV 2024-12 conditional novelty 5.0 of 10

A multimodal dataset and diffusion method that explicitly separates content from style in artistic images, reducing content leakage during stylization.
PyPotteryLens: An Open-Source Deep Learning Framework for Automated Digitisation of Archaeological Pottery Documentation
cs.CV 2024-12 conditional novelty 5.0 of 10

PyPotteryLens detects, segments, orients, and labels pottery drawings from archaeological PDFs using YOLO and EfficientNetV2, reporting above 96% precision and up to 20x faster processing.
ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
cs.AI 2024-12 conditional novelty 5.0 of 10

A human-scored benchmark shows that even GPT-4o averages below 4/5 correctness and all tested models struggle with scientific diagram prompts that combine spatial, numeric, and attribute requirements.
DetailCLIP: Injecting Image Details into CLIP's Feature Space
cs.CV 2022-08 unverdicted novelty 5.0 of 10

A patch-based fusion method extends CLIP to high-resolution images by retaining multi-scale details for improved class-prompted retrieval.

See all 78 Pith citations

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · cited by 78 Pith papers (see all)

[1]

Bowman et al

The KL weight β is increased from 0 to 6.6 over the ﬁrst5000 updates. Bowman et al. (2015) use a similar schedule based on the sigmoid function

work page 2015
[2]

Using a linear annealing schedule for this typically led to divergence

The relaxation temperature τ is annealed from 1 to 1/16 over the ﬁrst 150,000 updates. Using a linear annealing schedule for this typically led to divergence

work page
[3]

row, column, row, row

The step size is annealed from 1· 10−4 to 1.25· 10−6 over 1,200,000 updates. The decay schedules for the relaxation temperature and the step size are especially important for stability and successful optimization. We update the parameters using AdamW (Loshchilov & Hutter, 2017) with β1 = 0.9, β2 = 0.999, ϵ = 10−8, and weight decay multiplier 10−4. We use ...

work page 2017
[4]

Our model uses 128 gradient scales, one for each of its resblocks

Use per-resblock gradient scaling (Figure 4) instead of standard loss scaling. Our model uses 128 gradient scales, one for each of its resblocks. All of the gradient scales are initialized toM· 213, where M is the number of data-parallel replicas (i.e., the number of GPUs). In our setup, each grad scale is multiplied by 21/1000 at every parameter update w...

work page
[5]

In particular, store all gains, biases, embeddings, and unembeddings in 32-bit precision, with 32-bit gradients (including for remote communication) and 32-bit Adam moments

Only use 16-bit precision where it is really necessary for performance. In particular, store all gains, biases, embeddings, and unembeddings in 32-bit precision, with 32-bit gradients (including for remote communication) and 32-bit Adam moments. We disable gradient compression for these parameters (though PowerSGD would not make sense for 1D parameters li...

work page
[6]

For data-parallel training, we need to divide the gradients by the total number of data-parallel workers M

Avoid underﬂow when dividing the gradient. For data-parallel training, we need to divide the gradients by the total number of data-parallel workers M. One way to do this is to divide the loss by the per-machine batch size, and then divide the parameter gradients by M before summing them over the machines (using all-reduce). To save time and space, the gra...

work page 2019
[7]

Our training setup uses a combination of parameter sharding and gradient compression, as described in Section 2.5. During backpropagation, while recomputing the activations and computing the gradients for the current resblock, we prefetch the parameters for the preceding resblock using all-gather. Once each GPU has computed the gradient with respect to a ...

work page
[8]

Otherwise, we do nothing and proceed with backpropagation; a single nonﬁnite value in the gradient means that the entire update will be skipped, which happens about 5% of the time

If there are no nonﬁnite values in the result of the reduce-scatter (which could be caused by overﬂow in backpropagation or the reduce-scatter), we divide the result by the resblock’s gradient scale, and add it to the error buffer (i.e., the buffer used for error correction). Otherwise, we do nothing and proceed with backpropagation; a single nonﬁnite val...

work page

Show all 23 references

[9]

Both the P and Q matrices are stored in 1-6-9 format and have their values scaled by predetermined constants, as discussed in Section D

Once the reduce-scatter operations for the resblock have ﬁnished, we schedule the operations to compute theP matrices from the errors buffers and the Q matrices, whose values are ﬁxed at the start of training (see Section 2.5). Both the P and Q matrices are stored in 1-6-9 for...
[10]

This all-reduce is carried out in the 1-6-9 format, using a custom kernel

Once each GPU has computed the P matrices for the parameter shards in a resblock, they are averaged with the P matrices from the GPUs with the same ordinal on all other machines, using a single, grouped all-reduce operation. This all-reduce is carried out in the 1-6-9 format, ...
[11]

We use a custom Householder orthogonalization kernel rather than Gram-Schmidt, as we found the latter to be numerically unstable

Once the all-reduce operation for the P matrices for a resblock have ﬁnished, we orthogonalize the columns of the resulting matrices. We use a custom Householder orthogonalization kernel rather than Gram-Schmidt, as we found the latter to be numerically unstable. We also add ϵ...
[12]

Zero-Shot Text-to-Image Generation

Once the P matrices for a resblock have been orthogonalized, we schedule the operations to compute the new Q matrices from the error buffers and the P matrices. Zero-Shot Text-to-Image Generation
[13]

As in step (4), we clamp all inﬁnities in the results of the all-reduce to the maximum value of the 1-6-9 format, retaining the sign

Once the new Q matrices for a resblock have been computed, we schedule another grouped all-reduce, similar to what we did for the P matrices. As in step (4), we clamp all inﬁnities in the results of the all-reduce to the maximum value of the 1-6-9 format, retaining the sign. T...
[14]

Section D explains why we use 32-bit precision for these parameters and their gradients

The gradients for all parameters that are not compressed are grouped together into a single, 32-bit precision all-reduce. Section D explains why we use 32-bit precision for these parameters and their gradients
[15]

Once all GPUs on a machine have ﬁnished steps (7) and (8) for every resblock in the model, the values of theP and Q matrices for the same parameter shard on all machines will be identical. We then compute the global gradient norm, which is the sum of two quantities: (a) the su...
[16]

While computing the global norm, we also synchronize the information from step (2) about which parameter shard gradients contained nonﬁnite values after the reduce-scatter. After doing this, we have two pieces of information for each parameter shard: (a) whether its error buff...
[17]

Like backpropagation, the parameter updates proceed resblock-by-resblock

Once all of the all-reduces have ﬁnished and the global norm has been computed, we can apply the parameter updates. Like backpropagation, the parameter updates proceed resblock-by-resblock. The ﬁrst step is to compute the decompressed gradients by forming the product P Qt for ...
[18]

local” gradient averaged over the GPUs on the machine using reduce-scatter, and the “remote

The second step is the update to the error buffers. First, we use the results from step (10) to check if the P and Q matrices for a given parameter shard contain only ﬁnite values. If this is the case, then we divide the decompressed gradient by the total number of machines, a...
[19]

We also note the following important optimizations:

The parameter shards whose gradients are not compressed are updated separately. We also note the following important optimizations:
[20]

For example, while we are running step (2) for resblock i, we can proceed to steps (3)–(8) for all resblocks j > i

There are several opportunities for overlap between compute and communication in the above steps. For example, while we are running step (2) for resblock i, we can proceed to steps (3)–(8) for all resblocks j > i . Exploiting opportunities for overlap is necessary to achieve g...
[21]

For example, we only prefetch the parameters from the preceding resblock when the reduce-scatter operations have ﬁnished for the current one

We throttle speciﬁc operations that are liable to exhaust all available memory. For example, we only prefetch the parameters from the preceding resblock when the reduce-scatter operations have ﬁnished for the current one. Otherwise, we risk running out of memory by holding on ...
[22]

The former inﬂuences the bandwidth analysis, which we present in Section E.1

There are two places in the implementation where the transposition matters: (a) the choice of shard axis for the MLP matrices and (b) whether we compute the low-rank factorization for a gradient or its transpose. The former inﬂuences the bandwidth analysis, which we present in...
[23]

the exact same cat on the top as a sketch on the bottom

In step (12) above, we note that setting the error buffers to zero too often can cause performance regressions. We wanted to avoid doing this when resuming training from a checkpoint, which happens more frequently for larger jobs as it is likely that a machine will periodicall...

Pith tools

Reviewed May 13, 2026 · model on record in the stance chip above.

[1] [1]

Bowman et al

The KL weight β is increased from 0 to 6.6 over the ﬁrst5000 updates. Bowman et al. (2015) use a similar schedule based on the sigmoid function

work page 2015

[2] [2]

Using a linear annealing schedule for this typically led to divergence

The relaxation temperature τ is annealed from 1 to 1/16 over the ﬁrst 150,000 updates. Using a linear annealing schedule for this typically led to divergence

work page

[3] [3]

row, column, row, row

The step size is annealed from 1· 10−4 to 1.25· 10−6 over 1,200,000 updates. The decay schedules for the relaxation temperature and the step size are especially important for stability and successful optimization. We update the parameters using AdamW (Loshchilov & Hutter, 2017) with β1 = 0.9, β2 = 0.999, ϵ = 10−8, and weight decay multiplier 10−4. We use ...

work page 2017

[4] [4]

Our model uses 128 gradient scales, one for each of its resblocks

Use per-resblock gradient scaling (Figure 4) instead of standard loss scaling. Our model uses 128 gradient scales, one for each of its resblocks. All of the gradient scales are initialized toM· 213, where M is the number of data-parallel replicas (i.e., the number of GPUs). In our setup, each grad scale is multiplied by 21/1000 at every parameter update w...

work page

[5] [5]

In particular, store all gains, biases, embeddings, and unembeddings in 32-bit precision, with 32-bit gradients (including for remote communication) and 32-bit Adam moments

Only use 16-bit precision where it is really necessary for performance. In particular, store all gains, biases, embeddings, and unembeddings in 32-bit precision, with 32-bit gradients (including for remote communication) and 32-bit Adam moments. We disable gradient compression for these parameters (though PowerSGD would not make sense for 1D parameters li...

work page

[6] [6]

For data-parallel training, we need to divide the gradients by the total number of data-parallel workers M

Avoid underﬂow when dividing the gradient. For data-parallel training, we need to divide the gradients by the total number of data-parallel workers M. One way to do this is to divide the loss by the per-machine batch size, and then divide the parameter gradients by M before summing them over the machines (using all-reduce). To save time and space, the gra...

work page 2019

[7] [7]

Our training setup uses a combination of parameter sharding and gradient compression, as described in Section 2.5. During backpropagation, while recomputing the activations and computing the gradients for the current resblock, we prefetch the parameters for the preceding resblock using all-gather. Once each GPU has computed the gradient with respect to a ...

work page

[8] [8]

Otherwise, we do nothing and proceed with backpropagation; a single nonﬁnite value in the gradient means that the entire update will be skipped, which happens about 5% of the time

If there are no nonﬁnite values in the result of the reduce-scatter (which could be caused by overﬂow in backpropagation or the reduce-scatter), we divide the result by the resblock’s gradient scale, and add it to the error buffer (i.e., the buffer used for error correction). Otherwise, we do nothing and proceed with backpropagation; a single nonﬁnite val...

work page