{"id":"a973f3c7-7f85-4d30-b505-f0ac142aff0e","arxiv_id":"2412.01827","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RandAR shows a decoder-only transformer can generate images in arbitrary token orders with quality close to raster order, using position instruction tokens, while unlocking parallel decoding and zero-shot editing.","lead":"A picture-generation model was trained to draw image pieces in random order instead of the usual top-to-bottom scan, using a small position hint before each piece. It matches the quality of ordered generation while enabling faster sampling and image editing tasks the model was not specifically trained for.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Parallel decoding conditions each sampled token on later position-instruction tokens that never appear in the teacher-forced training distribution, so the claim that RandAR samples the Eq. (4) distribution at 2.5x speed is not established by the reported FID/latency numbers.","rationale":"The reader's weakest_assumption concerns permutation coverage in Sec. 3.2 Training and generalization to unseen orders. I agree that coverage is a genuine concern and the paper provides no coverage or distributional analysis. However, the more precise load-bearing concern is that even the standard reported inference protocol does not sample the trained distribution: parallel decoding conditions on consecutively appended position-instruction tokens before the corresponding image tokens are generated, which never occurs in the teacher-forced training data. The paper's Sec. 3.3 is explicit about this procedure, and its rearrangement of the sequence happens only after the forward pass, so the conditioning at prediction time is different from Eq. (4). The low FID in Fig. 4(b) suggests the model has learned a robust random-order predictor, which is internally consistent and partially supports the zero-shot claims. But the claim that parallel decoding reduces steps 'without sacrificing generation quality' of the same distribution is not established, because the exact distribution being sampled from is not Eq. (4). The other issues raised by the reader — overlap with RAR [58] and the 'only decoder-only' claim — are about framing and prior disclosure, not the core correctness argument. They affect novelty but not the main technical validity. The comparable-FID and latency numbers are supported by the experiments, so the verdict should remain CONDITIONAL rather than moving to REJECT or UNVERDICTED. The condition is a controlled comparison between the exact training interleaving and the parallel schedule on matched permutations, plus an inference-order locality test. The paper's own Limitation section does not mitigate this concern because it only concedes data efficiency, not the distributional mismatch of the parallel and zero-shot inference procedures.","tokens_in":20555,"tokens_out":2706,"duration_ms":30281,"concrete_test":"Compare FID-50K and, if measurable, per-token cross-entropy across three inference protocols on the same RandAR-XL checkpoint: (a) the paper's 88-step parallel schedule, (b) 256-step sequential decoding in the exact training interleaving with uniformly random permutations (1 token per step), and (c) a 'clustered-prefix' schedule where the first 25% of tokens are randomly chosen but spatially concentrated at the image border, followed by uniformly random remaining tokens. If (a) and (b) match closely, the parallel schedule is a faithful approximate sampler and the 2.5x claim is supported. If (b) differs substantially from (a), the reported parallel-decoding quality rests on the off-distribution conditioning shortcut.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that RandAR, trained by Eq. (4) on randomly permuted but strictly interleaved teacher-forced sequences, performs next-token prediction at any location given previously generated tokens, and that parallel decoding (Sec. 3.3) is a zero-shot acceleration of that trained distribution. In training, every image token x is conditioned on a history whose visible token set is exactly the tokens preceding it in the sampled permutation, and each such history is immediately followed in the sequence by that token's position instruction token. During parallel decoding, the model is instead presented with multiple position instruction tokens appended consecutively, e.g., [P_n, P_{n+1}] in Eq. (6), and must predict x_n and x_{n+1} from a history that includes the 'future' instruction token P_{n+1}, which never occurs before x_n in the training distribution. After sampling, the sequence is rearranged into the training interleaving, but the conditioning for the earlier token was computed with P_{n+1} already in its context, i.e., with information about the position of a not-yet-generated token. The low FID at 88 steps in Fig. 4(b) and Table 4 shows this mismatch is empirically benign, but it means those numbers validate an approximate parallel-inference procedure, not the exact distribution of Eq. (4). The same procedure is then used for the zero-shot applications: inpainting/outpainting condition on arbitrary visible token sets, and resolution extrapolation samples from a 4x token grid. The paper's coverage argument (Sec. 3.2 Training, 3x10^8 of 8x10^506 permutations) applies to the teacher-forced training signal, not to these inference-time conditioning distributions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"RandAR trains a decoder-only LLaMAGen-style transformer on randomly permuted VQGAN token sequences, inserting a shared 'position instruction token' (rotated by 2D-RoPE) before each image token and supervising prediction of that image token from the instruction token. On ImageNet 256x256, RandAR-XL reaches FID 2.25 at 88 parallel-decoding steps versus 2.16 for an identically trained raster-order counterpart at 256 steps, which supports the paper's 'comparable quality' claim. The paper further reports 2.5x latency reduction through parallel decoding and demonstrates zero-shot inpainting, outpainting, 512x512 resolution extrapolation, and second-pass bi-directional feature encoding. It includes ablations of the position-instruction design (Table 2), inference-time orders (Table 3), and an honest Limitations section noting that random-order learning is less data-efficient than fixed-order training.","tokens_in":20971,"tokens_out":8870,"duration_ms":85770,"significance":"If the claims hold, RandAR is a conceptually useful step: it gives decoder-only autoregressive image models the flexibility usually associated with masked or bi-directional models while retaining plain next-token training and KV-cache compatibility. The central empirical comparison is well designed: the raster-order counterpart is trained with the same architecture and setup, FID-50K follows the standard ADM protocol, and latency is measured on the same hardware. The paper also provides pseudo-code for training and parallel decoding, and several negative results are reported honestly (e.g., the raster-order model collapses under the second-pass encoding in Table 5, and the 'Merge' position design hurts parallel decoding in Table 2). The main reservations are that the parallel-decoding procedure is described as exact when it is in fact an approximation, the 'arbitrary order' claim rests on generalization from a tiny fraction of the permutation space without direct analysis, and some headline zero-shot applications are evaluated only qualitatively. The work is worth publishing after these points are addressed.","major_comments":[{"comment":"Parallel decoding introduces a train/inference mismatch that the paper does not acknowledge. In training (Eq. (4)) every image token is immediately preceded by its own position instruction token, and consecutive position instruction tokens never occur. In Eq. (6), however, the model is given the context [..., P_n, P_{n+1}] and asked to predict x_n from a history that already contains the future instruction token P_{n+1}. The rearrangement in Eq. (7) fixes the final sequence format but does not change the conditioning under which x_n was sampled. Thus the parallel procedure is not an exact sampler of the Eq. (4) distribution; the small FID gap in Fig. 4(b) and Table 1 shows that the approximation is empirically benign, but the text should say so explicitly and, ideally, compare one-token-at-a-time decoding with the parallel schedule at equal step budgets to quantify the approximation error.","section":"Sec. 3.3, Eq. (6)-(7), Algorithm B"},{"comment":"The paper acknowledges that 300 epochs cover at most 3x10^8 of roughly 8x10^506 possible 256-token permutations. All of the zero-shot claims (inpainting, outpainting, resolution extrapolation, second-pass encoding) involve conditioning patterns and sequence lengths that are extremely unlikely to have appeared in training, so the claim of 'arbitrary orders' needs direct support. Table 3 tests a handful of hand-picked inference orders, but it does not relate performance to permutation coverage, order statistics, or the distance between an inference order and the training distribution. I would like to see either an experiment that varies the number or type of permutations seen during training and measures FID and downstream zero-shot quality, or an explicit caveat that 'arbitrary' means 'orders statistically similar to the uniform random training distribution.' Without one of these, the zero-shot framing overreaches.","section":"Sec. 3.2 'Training'; Table 3"},{"comment":"Resolution extrapolation is presented as a headline capability, but the only evidence is qualitative. The paper itself concedes in Sec. 4.4.3 that the model struggles with intricate structures and small high-frequency patterns. Since the claim is 'zero-shot generalization to 512x512', some quantitative measure is needed before this contribution can be assessed: for example, FID of the 512x512 outputs against appropriately scaled real data, a distribution-distance metric, or a perceptual/user study relative to the 256x256 baseline. As written, the reader cannot tell how often the extrapolation succeeds or how severe the observed failures are.","section":"Sec. 4.4.3, Fig. 6, Sec. G.3"},{"comment":"The caption of Table 1 states that RandAR is 'the only decoder-only method capable of generating images in random token orders,' yet the same table and related work cite RAR [58], whose title is 'Randomized Autoregressive Visual Generation.' The paper should clarify the precise technical difference between RandAR and RAR (is it the position-instruction-token mechanism, the use of full random permutations, or something else?) and soften the uniqueness claim if RAR already performs random-order decoding. As written, the novelty claim is not supported by the paper's own references.","section":"Table 1 caption; Related Work [58]"}],"minor_comments":[{"comment":"The notation is inconsistent: Eq. (2) drops the last image token, but Eq. (3) ends with P_N without the corresponding x_N, while Algorithm A and Eq. (4) use all N tokens. Please fix the sequence definition so that the indices and sequence lengths match the training code.","section":"Sec. 3.2, Eq. (2)-(3)"},{"comment":"The word 'casual' appears in the 'Casual AR' table headings and in 'full casual attention' (Sec. 4.4.2, Fig. 5); this should be 'causal' throughout.","section":"Table 1 and Fig. 5"},{"comment":"The formula for spatial contextual guidance contains an extra unmatched parenthesis in the term -(e_theta(x^phi_{1:n}, c)); please correct the typesetting.","section":"Supplementary, Eq. (A)"},{"comment":"The description of the KV-cache rewrite is terse and relies on an example inside a code comment. A short formal invariant stating that the effective KV-cache always matches the training-time interleaved format would help readers verify the correctness of the cache manipulation.","section":"Sec. 3.3 and Algorithm B"},{"comment":"The text says '300 epochs ... 360K iterations' with batch size 1024; for ImageNet this is approximately 375K iterations, so the stated number should be reconciled with the actual training schedule.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The relationship to RAR [58] should be resolved before acceptance; the 'only decoder-only method' claim is likely to be challenged by reviewers familiar with that work. The editor may also wish to decide whether qualitative-only evidence for resolution extrapolation meets the journal's bar for a headline capability claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper with a real design contribution, but the novelty story needs rewriting because RAR already did decoder-only random-order generation and the paper never grapples with that overlap. The genuinely new pieces are the position-instruction token (one shared embedding rotated with 2D RoPE), the parallel-decoding schedule, and the hierarchical resolution-extrapolation recipe. The training setup is well specified and the self-trained raster-order counterpart is the right control; FID 2.25 vs 2.16 at XL size supports the central comparable-quality claim.\n\nWhat the paper does well: it validates the main claim with the right comparison, it is honest in the limitations section about random-order training being harder, and the bi-directional feature extraction result (second pass improves SPair71k from 22.1 to 31.3 while the raster model collapses) is a concrete empirical finding worth reporting. The pseudo-code for training and parallel decoding is detailed enough to reproduce the method.\n\nSoft spots, in order of importance. First, Table 1 lists RAR [58] but the text never discusses what RandAR adds over it, and the abstract's 'Unlike previous decoder-only AR models' is false as written. This needs fixing before publication. Second, several headline capabilities are only qualitative: inpainting and outpainting have no quantitative metric, and resolution extrapolation has no baseline comparison. The paper's own Fig. 6 admits failures on structured objects. Third, the parallel decoding story has a real train/inference mismatch. In training each image token is conditioned on a history ending with its own position instruction; in parallel decoding the model sees two instruction tokens back-to-back and predicts two tokens at once. The FID numbers show the mismatch is empirically benign, but the claim that parallel decoding samples the Eq. (4) distribution at 2.5x speed is not literally established. That is a wording fix, not a fatal flaw.\n\nThe permutation-coverage concern from the stress test does not land as a load-bearing flaw. Only a tiny fraction of the 256! orders is seen, but training samples fresh random permutations each step, and the zero-shot results are direct evidence of transfer to unseen orders. More analysis would be nice, but the empirical case is reasonable.\n\nWho this is for: people working on decoder-only visual generation, especially sampling-order flexibility, parallel decoding, and resolution extrapolation. It deserves serious refereeing. The overlap with RAR must be resolved and the zero-shot claims quantified, but the core mechanism and controls are sound.","headline":"Solid empirical design with real applications, but novelty is overstated: RAR already did random-order decoder-only generation, and the paper needs to confront that overlap.","tokens_in":21487,"tokens_out":2696,"would_cite":true,"duration_ms":27557,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decoder-only transformer can generate images in arbitrary token orders without loss of quality, if each image token is preceded by a position instruction token telling it where in the image it lives.","keywords":["random-order autoregressive generation","decoder-only transformer","position instruction token","image generation","parallel decoding","zero-shot inpainting","outpainting","resolution extrapolation"],"falsifier":"Evaluate a trained RandAR model on inference orders deliberately constructed to be far from the uniform random-order training distribution (for example, fixed adversarial permutations, or orders whose first tokens all come from one corner of the image) and compare FID with the reported random-order FID of 2.25.","tokens_in":1721,"feed_emoji":"🎨","tokens_out":2077,"duration_ms":93677,"temperature":0.7,"pith_summary":"This paper asks whether a GPT-style decoder-only image generator really needs the fixed raster order (top-left to bottom-right) that nearly all such models use. It claims the answer is no: inserting a small position instruction token before each image token lets the same architecture be trained to predict tokens in any order, and the resulting model matches its raster-order counterpart on ImageNet 256x256 generation (RandAR-XL FID 2.25 vs 2.16). The payoff is a set of zero-shot abilities that fixed-order models lack, including parallel decoding for roughly 2.5x faster sampling, inpainting, outpainting, resolution extrapolation to 512x512, and bi-directional feature extraction. The paper positions this as evidence that the order bias is not a necessary cost of decoder-only generation, and that plain next-token prediction can model 2D images with bidirectional context.","feed_headline":"One token per patch lets image models generate in any order","feed_subtitle":"One position token per image patch unlocks parallel decoding, inpainting, outpainting, and resolution extrapolation.","key_machinery":"The position instruction token is the load-bearing object: a single shared learnable embedding rotated by 2D rotary position embeddings (2D-RoPE) to the coordinates $(h_i, w_i)$ of the next image token, and inserted immediately before that token in the shuffled sequence. It tells the causal transformer where in the image the token being predicted lives, so the model can be trained on fully random permutations of the 256 tokens instead of a fixed order. Because the context and target positions are arbitrary, the same mechanism directly supports parallel decoding (several position instructions appended at once), inpainting and outpainting (any visible tokens as context, any target positions to fill), and two-pass bidirectional encoding (run the sequence twice and read the second-pass features).","core_discovery":"The paper's central claim is that random-order next-token prediction, not a predefined raster sequence, is what makes decoder-only visual autoregressive models rigid, and that removing the order constraint is possible without sacrificing generation quality. RandAR does this by training a standard causal transformer on randomly permuted image-token sequences with a position instruction token before each token, making every prediction condition on an arbitrary subset of the image. On ImageNet 256x256, RandAR-XL reaches FID 2.25 with parallel decoding in 88 steps, essentially matching its raster-order counterpart's 2.16, and then reuses the same weights for zero-shot inpainting, outpainting, high-resolution synthesis, and bidirectional encoding. The paper interprets these results as showing that a plain decoder-only transformer can acquire the bi-directional context modeling that previously required masked encoder-decoder designs.","pith_inferences":["If the permutation-generalization assumption holds, the same position-instruction mechanism could be dropped into existing decoder-only multimodal models that currently hard-code raster order for images, giving them the same zero-shot editing and parallel-decoding abilities without retraining from scratch.","The two-pass bidirectional encoding result suggests random-order decoder-only transformers could serve as generative vision encoders, but the paper only evaluates linear probing and semantic correspondence; broader transfer tasks such as detection and segmentation remain untested.","A direct test of the mechanism would be to train RandAR on a restricted family of permutations (for example, only hierarchical or locality-preserving orders) and measure how much of the zero-shot capability disappears; the paper does not vary permutation coverage during training.","The resolution-extrapolation recipe of generating even coordinates first and then filling details with NTK-style high-frequency RoPE looks like a general coordinate-sampling schedule that might transfer to arbitrary aspect ratios or video frames, though the paper demonstrates it only for square 256-to-512 synthesis."],"forward_implications":["RandAR reaches FID 2.25 at 88 steps with parallel decoding, essentially matching its raster-order counterpart's 2.16, so random-order training costs little generation quality.","Parallel decoding cuts latency roughly 2.5x (from 16.8s to 6.6s for a 1.4B model on A100), because the model can predict multiple tokens in one forward pass.","The same 256x256-trained model performs zero-shot inpainting, outpainting to 256x1024, and resolution extrapolation to 512x512 without fine-tuning.","Passing the token sequence through the causal transformer twice yields bi-directional features that improve SPair-71k semantic correspondence (PCK 31.3 vs 24.5 for the raster-order model) and ImageNet linear probing.","Inference-order ablations show fully random orders outperform fixed orders (random FID 2.25 vs raster FID 4.82), indicating the model learns to exploit broad context rather than memorize a sequence."],"supporting_citations":[{"why":"Supplies the base decoder-only architecture, training recipe, and ImageNet VQGAN tokenizer; the raster-order counterpart is this same setup without position instruction tokens.","marker":"[45]"},{"why":"MaskGIT demonstrates masked random-order generation with bi-directional attention, giving RandAR its target capabilities of parallel decoding and inpainting.","marker":"[5]"},{"why":"XLNet contributes permutation language modeling and the target-aware representation idea that motivates inserting position instruction tokens.","marker":"[55]"},{"why":"2D-RoPE from RoFormer is used both as the relative positional encoding inside attention and to rotate the shared position instruction embedding to image coordinates.","marker":"[44]"},{"why":"VQGAN provides the discrete image tokenizer and the fixed generation orders (raster, spiral-in, spiral-out, z-curve, subsample, alternate) used in the inference-order ablation.","marker":"[10]"},{"why":"MAR is the bi-directional autoregressive baseline compared for generation quality and for parallel-decoding latency without KV-cache support.","marker":"[25]"},{"why":"RAR is the concurrent randomized-autoregressive generation model listed among the causal AR baselines that RandAR compares against in Table 1.","marker":"[58]"}],"fun_headline_variants":["Position tokens let decoder-only models draw any image order","Random-order training gives decoder-only models new image abilities","No fixed order: decoder-only model handles inpainting, outpainting","RandAR: train on shuffled tokens to get zero-shot inpainting and more","Decoder-only AR with position tokens enables arbitrary-order generation"],"cache_read_input_tokens":23552,"weakest_assumption_plain":"The load-bearing premise is that training on only about $3\\times10^8$ of the roughly $8\\times10^{506}$ possible $256$-token orders teaches the model to generate well on arbitrary unseen orders at inference.","fun_headline_variants_meta":{"raw":{"variants":["Position tokens let decoder-only models draw any image order","Random-order training gives decoder-only models new image abilities","No fixed order: decoder-only model handles inpainting, outpainting","RandAR: train on shuffled tokens to get zero-shot inpainting and more","Decoder-only AR with position tokens enables arbitrary-order generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001084,"raw_usage":{"total_tokens":4512,"prompt_tokens":908,"completion_tokens":3604,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":3517}},"tokens_in":524,"tokens_out":3604,"duration_ms":26996,"temperature":1.0,"reasoning_tokens":3517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:54:21.394791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate a trained RandAR model on inference orders deliberately constructed to be far from the uniform random-order training distribution (for example, fixed adversarial permutations, or orders whose first tokens all come from one corner of the image) and compare FID with the reported random-order FID of 2.25.","supporting_citations":[{"cited_title":"Maskgit: Masked generative image transformer","cited_arxiv_id":null,"evidence_quote":"MaskGIT demonstrates masked random-order generation with bi-directional attention, giving RandAR its target capabilities of parallel decoding and inpainting."},{"cited_title":"Carbonell, Ruslan Salakhutdinov, and Quoc V","cited_arxiv_id":null,"evidence_quote":"XLNet contributes permutation language modeling and the target-aware representation idea that motivates inserting position instruction tokens."},{"cited_title":"Autoregressive image generation without vec- tor quantization","cited_arxiv_id":null,"evidence_quote":"MAR is the bi-directional autoregressive baseline compared for generation quality and for parallel-decoding latency without KV-cache support."}],"review_version":1}