{"id":"7fef273c-5eac-4409-8c93-0d6e76965da0","arxiv_id":"2502.04896","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A joint image-video generation model family reports state-of-the-art benchmark scores using rectified flow transformers, with all key evidence self-reported and no artifacts released.","lead":"Goku is a family of large transformer models for generating images and videos from text, trained with rectified flow on a curated dataset of 196 million text-media pairs. The paper claims top benchmark scores, including 84.85 on VBench, but releases no weights, code, or data that would let others verify them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Goku's headline VBench 84.85 cannot be checked: Appendix A lists no generation protocol for VBench and no decontamination against evaluation prompts, and the 0.58-point margin over CausVid leaves the top-leaderboard claim vulnerable to protocol drift or prompt rewriting.","rationale":"The reader's weakest assumption and my own reading converge on the same point: the headline benchmark numbers are self-reported with incomplete evaluation protocols, and the strongest claim, top position on VBench, depends on an 84.85 score whose margin over the nearest competitor is small. I considered the UCF-101 result as a potentially sharper concern: Table 6 declares SOTA at 128x128 while the same model at 256x256 scores 246.17, above PixelDance's 242.82, so the claimed SOTA appears to come from choosing a resolution not used by baselines. That is a real protocol inconsistency and it supports the broader worry. However, the paper's own introduction and abstract place more weight on VBench and GenEval/DPG, so the VBench configuration gap is the most load-bearing issue. It is addressable: a precise protocol, decontamination check, and released artifacts would settle it. Since the reader already reached CONDITIONAL for these reasons, my recommendation is UNCHANGED rather than a new verdict.","tokens_in":22910,"tokens_out":6435,"duration_ms":64420,"concrete_test":"Release the exact VBench evaluation recipe (prompt set, original versus rewritten; per-prompt video count; resolution; duration; fps; sampler; steps; CFG; seeds) and rerun the official VBench suite on the released Goku-8B checkpoint with no LLM prompt rewriting and with the same recipe used for the baseline numbers. If the overall score falls below 84.27, or if per-dimension values in Table 8 shift by more than about 1 point, the top-leaderboard claim is not established. As a secondary check, compute prompt n-gram overlap between the training captions and the VBench prompt set to rule out direct contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim rests on VBench 84.85 (Table 7), but the paper never specifies how this number was produced. Appendix A's VBench paragraph only describes the benchmark; it omits resolution, duration, frame rate, number of samples per prompt, sampler, sampling steps, classifier-free guidance, seed, and whether prompts were rewritten. VBench scores are sensitive to these settings, and the paper's margin over CausVid (84.27) is only 0.58 points, so a plausible protocol difference or prompt-rewriting step could flip the ranking. The same section also does not state whether training data was deduplicated against the public VBench/GenEval/DPG prompt sets; the data pipeline (Section 4) draws from LAION, Panda-70M, InternVid, and internet sources, which can overlap benchmark prompts. In addition, the paper's other SOTA evidence shows the same pattern: the abstract's GenEval 0.76 is obtained with ChatGPT-4o prompt rewriting (Table 5), and the UCF-101 SOTA in Table 6 uses a different resolution (128x128) than most baselines. None of this proves the score is wrong, but the load-bearing empirical claim is currently unverifiable and would not survive a changed protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Goku, a family of rectified-flow Transformer models for joint text-to-image and text-to-video generation, with 2B and 8B parameter variants. It describes a multi-stage data curation pipeline (filtering, captioning, balancing), a joint image-video VAE, full-attention transformer architecture with 3D RoPE and QK normalization, and a large-scale training infrastructure. The central empirical claims are that Goku-T2I achieves 0.76 on GenEval and 83.65 on DPG-Bench, and that Goku-T2V achieves 84.85 on VBench, placing it first on the VBench leaderboard as of 2025-01-25. The paper also reports a zero-shot UCF-101 FVD of 217.24 at 128x128 resolution.","tokens_in":23213,"tokens_out":2873,"duration_ms":27550,"significance":"If the stated results are reproducible under comparable protocols, Goku would be a strong joint image-video generation system and a useful reference point for scaling rectified-flow transformers. The paper's strengths include a detailed account of the data curation pipeline, the joint VAE and full-attention design, the multi-stage training recipe, and the infrastructure engineering (sequence parallelism, fine-grained activation checkpointing, fault tolerance, ByteCheckpoint). These engineering contributions are potentially valuable to the community. However, the headline state-of-the-art claims rest on benchmark scores whose measurement protocols are incompletely specified, and some of the comparisons are not apples-to-apples (prompt rewriting, resolution differences, and possibly evaluation-prompt overlap with training data). The paper would be acceptable only after these evaluation issues are resolved.","major_comments":[{"comment":"The VBench evaluation protocol is not specified. Appendix A's VBench paragraph only describes the benchmark suite; it omits the generation resolution, duration, frame rate, number of samples per prompt, sampler, sampling steps, classifier-free guidance, random seeds, and whether prompts were rewritten before generation. Since Goku's margin over CausVid is only 0.58 points (84.85 vs. 84.27), the top-leaderboard claim is sensitive to protocol differences. Please provide the full VBench configuration, including prompt handling and the exact number of generated videos per dimension.","section":"Appendix A, Table 7"},{"comment":"The GenEval comparison is not apples-to-apples. The headline 0.76 score is obtained after rewriting GenEval prompts with ChatGPT-4o, while the baseline scores in the same table are reported without this protocol (except for the †-marked DALL-E 3 and Emu3 entries). Prompt rewriting can systematically improve alignment scores, so the comparison is potentially biased. Please report Goku-T2I on the original GenEval prompts alongside baselines on the same prompts, and separately report rewritten-prompt scores for both Goku and baselines under an identical rewriting protocol.","section":"Table 5, Section 5.1"},{"comment":"The UCF-101 zero-shot evaluation uses captions generated by Tarsier-34B (Wang et al., 2024a), and the training captioning pipeline uses Tarsier2 (Yuan et al., 2025). This creates a risk that the evaluation prompts are drawn from the same model family used for training captions, which may favor Goku over baselines. Additionally, the state-of-the-art FVD claim is made at 128x128 resolution while most baselines are evaluated at 256x256 or higher; lower resolution can artificially improve FVD. Please state whether the Tarsier-generated UCF-101 prompts overlap with training data, and report Goku's results at the same resolution and prompt set as the key baselines.","section":"Section 5.2, Table 6"},{"comment":"All headline benchmark numbers are reported as single runs without error bars, variance, or the number of repeated generations. For a state-of-the-art claim that is decided by a 0.58-point margin on VBench, run-to-run variability could change the ranking. Please provide multiple seeds or at least a variance estimate for the core benchmark scores, or otherwise justify that the differences are statistically significant.","section":"Tables 5, 7, 8"},{"comment":"The paper does not state whether the training data was deduplicated against the public GenEval, DPG-Bench, or VBench prompt sets. The data pipeline draws from LAION, Panda-70M, InternVid, OpenVid, and internet sources, and the benchmarks use public prompts; overlap could inflate scores. Please describe any decontamination procedure, or report which benchmarks, if any, were used to filter training data.","section":"Section 4.1, Section 5"}],"minor_comments":[{"comment":"The abstract states 'Goku achieves 0.76 on GenEval' without noting that this result uses ChatGPT-4o prompt rewriting; please add the qualifier for consistency with the table.","section":"Abstract and Table 5"},{"comment":"There is a typo: 'collet' should be 'collect'.","section":"Section 4.1"},{"comment":"The table would be clearer if the best result at a resolution comparable to each baseline were separated from the best overall; otherwise the reader may conflate the 128x128 FVD with a like-for-like comparison.","section":"Table 6"},{"comment":"The names 'T2I-Compbench' and 'T2I-CompBench' are used inconsistently; please unify the capitalization.","section":"Appendix A"},{"comment":"The phrase 'an video-language model' should be 'a video-language model'.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems-oriented report with substantial engineering content, but its central SOTA claims are not yet verifiable because evaluation protocols are missing and several comparisons are not controlled. The issues are likely fixable with additional experimental detail, so I do not see a grounds for rejection at this stage, provided the authors can supply the missing protocol information and decontamination statements. I would also flag that the use of Tarsier-generated prompts for UCF-101 needs particular scrutiny, as the training captioner is from the same model family."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the engineering content is genuinely useful: the data curation pipeline (aesthetic, OCR, motion, DINO filtering with concrete thresholds), the multi-stage joint training strategy, and the infrastructure choices (Ulysses SP, FSDP hybrid shard, ByteCheckpoint) are described with enough specificity to be a practical recipe for anyone training a large multimodal generative model. Second, the headline numbers are not yet verifiable. The VBench 84.85 is presented as a leaderboard-topping result, but Appendix A gives no generation protocol—no resolution, duration, sampler, CFG, or whether prompts were rewritten. The margin over CausVid is 0.58 points, so a modest protocol change could flip the ranking. That is the load-bearing weakness. What the paper does well: it is an honest systems report. The pilot experiment comparing rectified flow against DDPM on ImageNet is a reasonable sanity check. The ablation of joint training vs. without is qualitative but plausible. The authors are transparent that the GenEval 0.76 comes with ChatGPT-4o prompt rewriting, even though they do not apply that protocol to most baselines. The abstract says 'pioneering step' for flow-based joint image-video generation, which overstates novelty—HunyuanVideo and others already use flow for video—but the specific combination of full attention, NaViT packing, 3D RoPE, and QK-norm on a joint VAE is a coherent, well-executed design. The soft spots are real but addressable. VBench protocol is unspecified; UCF-101 SOTA uses a lower resolution (128x128) than most baselines; the reported scores have no error bars; no code or weights are released. These are not fatal for a systems paper, but they mean the 'state-of-the-art' claim currently rests on self-reported, under-specified benchmarks. Who should read this: practitioners building large-scale image-video models will get concrete value from the data thresholds and training setup. The paper deserves serious refereeing—the engineering is substantial and the empirical claims are important, even if they need to be pinned down. My recommendation: send it to review, but require the authors to specify evaluation protocols in detail or release enough artifacts to make the benchmark numbers reproducible. If they cannot do either, the SOTA claims should be cut down to 'competitive' rather than 'leading.'","headline":"A serious industrial recipe for joint image-video generation with rectified flow, but the benchmark claims are under-specified and need protocol details or artifacts before they can be taken at face value.","tokens_in":679,"tokens_out":1948,"would_cite":true,"duration_ms":33549,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Goku claims a single flow transformer can lead both image and video generation benchmarks.","keywords":["rectified flow","text-to-video generation","text-to-image generation","joint image-video model","video VAE","flow transformer","VBench","GenEval"],"falsifier":"Run the released Goku-T2V checkpoint on VBench with the same video count, resolution, duration, sample steps, sampler, and classifier-free-guidance scale used for the strongest baselines, and rerun GenEval with every model using identical prompts (original and rewritten); if the margins disappear or reverse under a common protocol, the state-of-the-art claim fails. Checking whether any VBench or GenEval prompts appear in the described 160M image-text and 36M video-text training corpora would also settle contamination concerns.","tokens_in":22710,"feed_emoji":"🎬","tokens_out":5617,"duration_ms":45961,"temperature":0.7,"pith_summary":"This paper tries to establish that a single family of rectified-flow transformer models, trained jointly on images and videos, can set leading benchmark scores in both text-to-image and text-to-video generation. The authors report 0.76 on GenEval, 83.65 on DPG-Bench, and 84.85 on VBench, the last placing their text-to-video model at the top of the leaderboard as of January 25, 2025. If these results hold under comparable evaluation protocols, Goku would be evidence that image and video generation can share one latent space, one architecture, and one training run rather than needing separate specialized models. The paper's contribution is largely an engineering recipe: a data curation pipeline, a joint 3D VAE, multi-stage training, and infrastructure for long-sequence training.","feed_headline":"Goku flow transformer tops image and video benchmarks","feed_subtitle":"Joint model reports 84.85 on VBench, 83.65 on DPG-Bench, and 0.76 on GenEval.","key_machinery":"The carrying mechanism is rectified flow over a shared image-video latent space. A 3D joint VAE compresses video by $8 \\times 8 \\times 4$ and images by $8 \\times 8$; the transformer sees packed image and video tokens with full attention, positional information from 3D RoPE, and RMSNorm on query and key vectors to stabilize training. The flow objective is to predict the velocity $v_t = dx_t / dt$ along the linear interpolation, and multi-stage training lets the network first learn text-to-image semantics before adding temporal motion and then specializing per modality.","core_discovery":"Goku is a joint image-and-video generative model built on rectified flow: training samples interpolate linearly between noise and data, $x_t = t x_1 + (1-t) x_0$, and the transformer predicts the velocity field. Images and videos are encoded by a shared 3D VAE into one latent space, packed together in a full-attention transformer with 3D RoPE and Q-K normalization, and trained in three stages: text-to-image pretraining, joint image-video learning at increasing resolution, then modality-specific finetuning. On benchmarks, Goku-T2I reports 0.76 on GenEval (with ChatGPT-4o rewritten prompts) and 83.65 on DPG-Bench, and Goku-T2V reports 84.85 on VBench, which the paper states was the top leaderboard position as of 2025-01-25. The paper also claims rectified flow converges faster than DDPM in a controlled ImageNet-1K pilot, and that joint training improves video frame quality compared with video-only finetuning.","pith_inferences":["The authors' protocol for VBench is not stated; if their generation setup differs from that used for the commercial baselines, the leaderboard gap could shrink or reverse under a common protocol.","The GenEval 0.76 depends on rewriting prompts with ChatGPT-4o; applying rewriting uniformly to all baselines, or evaluating Goku without rewriting, would give a fairer comparison than the table's mixed protocol.","A testable extension suggested by the paper is training the same architecture with image and video data fully mixed from the start rather than staged, which would test whether the staged curriculum is load-bearing or merely convenient.","Beyond the paper, one could probe whether motion-score conditioning transfers to user-controlled motion strength in video editing tasks."],"forward_implications":["A single 8B transformer, trained on roughly 160M image-text and 36M video-text pairs, is claimed to outperform specialized commercial video models on VBench's overall score.","Joint image-video training is claimed to improve video frame photorealism relative to fine-tuning from the same image-pretrained weights on video only.","Rectified flow reaches a given FID-50K on ImageNet-256 with fewer training steps than DDPM in the paper's 1B pilot.","Captions that include a motion score give users a way to steer the amount of motion in generated videos.","Scaling from 2B to 8B parameters is claimed to reduce structural distortions such as malformed limbs and wheels."],"supporting_citations":[{"why":"Supplies the flow matching objective that Goku trains with.","marker":"Lipman et al., 2023"},{"why":"Defines rectified flow, the basis for Goku's velocity prediction and straight-path inference.","marker":"Liu et al., 2023"},{"why":"Provides the GenTron transformer block design that Goku extends with full attention and 3D RoPE.","marker":"Chen et al., 2024a"},{"why":"Introduces the diffusion transformer backbone that GenTron and Goku build on.","marker":"Peebles and Xie, 2023"},{"why":"Source of the jointly trained image-video VAE that gives Goku a shared latent space.","marker":"Yang et al., 2024c"},{"why":"Provides the VBench benchmark and leaderboard on which Goku-T2V reports 84.85.","marker":"Huang et al., 2024"},{"why":"Defines the GenEval benchmark used for the text-to-image alignment score.","marker":"Ghosh et al., 2024"},{"why":"Provides the DPG-Bench dense-prompt benchmark used for the 83.65 text-to-image score.","marker":"Hu et al., 2024"},{"why":"Supplies the LAION image-text data and aesthetic scoring models used in filtering.","marker":"Schuhmann et al., 2022"},{"why":"Introduces the Patch n' Pack sequence packing method used for joint variable-length image-video training.","marker":"Dehghani et al., 2024"}],"fun_headline_variants":["Goku flow model tops VBench 84.85, DPG-Bench 83.65","Goku rectified flow transformer hits top image and video benchmarks","Joint image-video Goku transformer hits top benchmarks with flow","Goku flow transformer: 84.85 VBench, 83.65 DPG-Bench","Flow-based Goku model beats benchmarks on image and video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported benchmark scores were produced under evaluation protocols comparable to those of the baselines and without training-data overlap with benchmark prompts, since the paper does not specify the VBench generation configuration and its headline GenEval score uses ChatGPT-4o prompt rewriting that was not applied to most baselines.","fun_headline_variants_meta":{"raw":{"variants":["Goku flow model tops VBench 84.85, DPG-Bench 83.65","Goku rectified flow transformer hits top image and video benchmarks","Joint image-video Goku transformer hits top benchmarks with flow","Goku flow transformer: 84.85 VBench, 83.65 DPG-Bench","Flow-based Goku model beats benchmarks on image and video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3351,"prompt_tokens":897,"completion_tokens":2454,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":2353}},"tokens_in":513,"tokens_out":2454,"duration_ms":16972,"temperature":1.0,"reasoning_tokens":2353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:04:05.081128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released Goku-T2V checkpoint on VBench with the same video count, resolution, duration, sample steps, sampler, and classifier-free-guidance scale used for the strongest baselines, and rerun GenEval with every model using identical prompts (original and rewritten); if the margins disappear or reverse under a common protocol, the state-of-the-art claim fails. Checking whether any VBench or GenEval prompts appear in the described 160M image-text and 36M video-text training corpora would also settle contamination concerns.","supporting_citations":[{"cited_title":"and Xie, S","cited_arxiv_id":null,"evidence_quote":"Introduces the diffusion transformer backbone that GenTron and Goku build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VBench benchmark and leaderboard on which Goku-T2V reports 84.85."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GenEval benchmark used for the text-to-image alignment score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LAION image-text data and aesthetic scoring models used in filtering."}],"review_version":1}