{"id":"7ed2d6cf-1e44-4568-a380-f78f691d9835","arxiv_id":"2504.14260","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"CrossWKV adapts RWKV-7's WKV state update into a cross-modal attention layer for diffusion text-to-image generation, but the reported benchmark results are explicitly labeled preliminary and in-progress.","lead":"This paper proposes CrossWKV, a cross-attention mechanism that adapts the linear-complexity RWKV-7 state model for text-to-image diffusion, reporting FID 2.88 and CLIP 0.33 on ImageNet 256x256. The paper's own evaluation section says these results are still in progress and preliminary, which undercuts the abstract's claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4 states the evaluation is 'currently in progress' while the abstract reports FID 2.88/CLIP 0.33; the reported numbers are explicitly preliminary, and the released artifact is only the generic rwkv7 layer, not CrossWKV.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I see: the paper's central empirical claim is not backed by completed evaluation. Section 4 explicitly says the evaluation is in progress and results are preliminary, directly contradicting the abstract's presentation of FID 2.88 and CLIP 0.33 as achieved. My independent read confirms this self-undermining structure and adds two supporting observations: the reported inference schedule is inconsistent between Section 3.5 and Section 4.1, and the only provided code artifact is a generic RWKV-7 layer, not the proposed CrossWKV mechanism. The dimensional ambiguity in Equation (1) is a separate internal weakness that would need to be resolved even if the evaluations were complete. Because the reader already recommended REJECT and my analysis reinforces that conclusion, no change to the verdict is needed. The proposed reproduction check would settle the concern: if the full pipeline and checkpoint are released and the numbers reproduce, the paper could be reassessed on its merits; if not, the headline claim should not be accepted.","tokens_in":9745,"tokens_out":3822,"duration_ms":35433,"concrete_test":"Download the linked repository at the cited version and verify whether it contains a CrossWKV module, DIR-7 training and evaluation scripts, and a saved ImageNet 256x256 checkpoint; if these exist, rerun the Section 4.2 protocol (10K generated samples, 250 DDPM steps, official clean-FID, and CLIP ViT-L/14 on 1K LAION prompts). If the repository contains only rwkv7.py, or if the reproduced FID differs from 2.88 by more than 0.1, the central claim is unverified. As a minimal analytical check, also re-derive Equation (1) under a single consistent row/column convention to confirm that the state update and output y_t are well-defined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the empirical headline: CrossWKV in DIR-7 matches DiT-class text-to-image quality with FID 2.88 and CLIP 0.33 on ImageNet 256x256. The load-bearing condition is that these numbers come from a completed, standardized evaluation. The paper itself denies this: the opening paragraph of Section 4 says the evaluation 'is currently in progress' and that 'preliminary results should be interpreted with caution.' Section 4.1 specifies a 250-step DDPM protocol with 10K samples, but no checkpoints, seeds, or generation details are provided, and the text elsewhere says inference uses a 50-step schedule (Section 3.5). Table 1 reports DIR-7-H at FID 2.88 with no error bars. The provided code link points to fla/layers/rwkv7.py, a generic RWKV-7 layer implementation, not to CrossWKV, the DIR-7 training pipeline, or the evaluation scripts. A secondary but independent issue reinforces the concern: Equation (1) is dimensionally unclear as written, since S_t is NxN, diag(w_t) is NxN, and v_t^T k_t is NxN only under one row/column convention, while the same symbols are treated as column vectors in Equation (3)'s k_i^T v_i term. Even if that convention can be repaired, the absence of a complete implementation makes the headline FID/CLIP numbers unreproducible from the available artifacts. The empirical assertion is therefore not supported by completed runs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CrossWKV, a cross-attention mechanism for the RWKV-7 state-based recurrent architecture, and integrates it into a text-to-image diffusion pipeline called DIR-7. It claims that CrossWKV performs global cross-modal fusion in a single pass with linear complexity and constant memory, and that DIR-7 achieves an FID of 2.88 and CLIP score of 0.33 on ImageNet 256x256, matching DiT-class models. The paper also claims that the non-diagonal input-dependent transition matrix of RWKV-7 enables representing functions beyond TC^0, including all regular languages and S5 permutation tracking. The authors state that the evaluation is in progress and provide a link to a generic RWKV-7 layer implementation.","tokens_in":10132,"tokens_out":2766,"duration_ms":25680,"significance":"If the empirical claims were substantiated, the paper could be significant: it proposes a plausible way to extend a linear-complexity recurrent architecture to cross-modal text-to-image generation, with potential efficiency advantages over transformer-based cross-attention. The connection to RWKV-7's expressivity is interesting, though it is not elaborated here. The paper is also transparent in stating that the evaluation is preliminary, which is commendable. However, the central empirical claim that DIR-7 matches state-of-the-art FID/CLIP scores is not supported by the submitted manuscript: the authors explicitly label the results as in-progress, no cross-attention code is provided, and the evaluation protocol has unresolved inconsistencies. The expressivity claims are imported from the RWKV-7 paper without any derivation or experiment specific to CrossWKV, so they do not add independent support.","major_comments":[{"comment":"The abstract and Table 1 present FID 2.88 and CLIP 0.33 as achieved results, but the opening paragraph of Section 4 states that the evaluation 'is currently in progress' and that the results are 'preliminary' and 'subject to further validation.' This is a direct contradiction of the paper's central claim. The empirical headline is load-bearing for the contribution, and the manuscript itself provides the grounds for not treating these numbers as established results. The code link in Section 4 points to https://github.com/TorchRWKV/flash-linear-attention/blob/dev/fla/layers/rwkv7.py, which is a generic RWKV-7 layer, not an implementation of CrossWKV or the DIR-7 training/evaluation pipeline; this makes the reported results unreproducible from the supplied artifact.","section":"§4 and Abstract"},{"comment":"Equation (1) is dimensionally inconsistent as printed. With S_t ∈ R^{N×N}, diag(w_t) ∈ R^{N×N}, and k_t, v_t ∈ R^N as defined, the term v_t^T k_t is a scalar, so the recurrence adds a scalar to an N×N matrix. If v_t^T k_t is intended to be an outer product or a rank-1 matrix, the notation must be changed (e.g., v_t k_t^T). A similar issue appears in Eq. (3), where k_i^T v_i is treated as a scalar while surrounded by matrix quantities. Because this recurrence is the core of the proposed mechanism, the missing convention makes the method ambiguous.","section":"§3.1, Eq. (1)"},{"comment":"The abstract and Introduction state that CrossWKV can represent functions beyond TC^0, including all regular languages and S5 permutation tracking, and that it performs 'board game modeling' such as Othello strategy optimization. No derivation, theorem, or experiment in this paper supports these claims; they are attributes of RWKV-7 from the cited work [Peng et al., 2025]. It is not shown that the cross-attention adaptation preserves the expressivity of the base architecture, nor is S5 or regular-language behavior tested for CrossWKV. These claims are part of the paper's positioning, so they need at least an argument or a reference to a specific result that directly covers the proposed modification.","section":"§1 and §4.8"},{"comment":"The inference and evaluation protocols are inconsistent. Section 3.5 and Section 4.1 state that inference uses a 50-step denoising schedule, while Section 4.2 says that FID is computed following Diffusion-RWKV's protocol with 250 DDPM steps. The reader cannot tell which schedule produced the numbers in Table 1, or whether the 50-step schedule is a separate experiment. Additionally, Table 1 reports no standard deviation or error bars, and Section 4.2 states that only 10K samples were generated, which is small for ImageNet FID at 256x256; these concerns compound the preliminary-status problem.","section":"§3.5, §4.1, §4.2"}],"minor_comments":[{"comment":"The last paragraph of the Introduction refers to 'Section ??' for limitations and future work; this reference is unresolved.","section":"§1"},{"comment":"The related work cites Stable Diffusion as [Blattmann et al., 2023], which is a stable video diffusion paper, while the actual Stable Diffusion paper is [Rombach et al., 2022]. The reference should be corrected.","section":"§2.1"},{"comment":"Table 2 is not referenced in the text; the ablation description would benefit from an explicit pointer to the table.","section":"§4.5"},{"comment":"The manuscript alternates between 'DIR-7' and 'CrossWKV' when describing the model being evaluated; the relationship between the module and the full system should be stated more clearly at each use.","section":"Throughout"},{"comment":"The code URL in the abstract points to the generic flash-linear-attention repository, not to a repository containing the CrossWKV implementation or the DIR-7 training/evaluation code; this should be clarified or corrected.","section":"Abstract"}],"recommendation":"reject","confidential_remarks":"The paper appears to be a preliminary or incomplete submission: the evaluation is self-described as in progress, the provided code does not implement the proposed method, and key claims are imported from another paper. These issues are not local presentation problems; they undermine the paper's central contribution as stated. If the authors complete the evaluation, release the actual code, and clarify the mathematical notation, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2504.14260. The new piece is legitimate: CrossWKV takes RWKV-7's WKV recurrence and turns it into a cross-modal layer by putting text in the receptance role, image in the key/value role, padding text to image length, and adding LoRA-gated decay. As a design, it's a plausible extension of the Diffusion-RWKV line, and if it held up it would give linear-scaling, constant-memory text-to-image conditioning. Credit where due: that is a real idea, and the authors name their baseline (Diffusion-RWKV) and the RWKV-7 kernel they build on.\n\nThe soft spots are not minor. The manuscript undermines its own headline: Section 4 opens by saying the evaluation is 'currently in progress' and that 'preliminary results should be interpreted with caution,' while the abstract reports FID 2.88 and CLIP 0.33 as achieved and the intro repeats the 'matching state-of-the-art' framing. Those numbers cannot be treated as established. The code link goes to fla/layers/rwkv7.py, the generic RWKV-7 layer, not to CrossWKV or the DIR-7 training/eval pipeline. So an independent group cannot reproduce the result, and the comparison table is not backed by completed runs.\n\nThen there are two smaller issues. Equation (1) is dimensionally sloppy as printed: S_t is NxN, and adding products of column vectors to a matrix times a diagonal matrix needs conventions that are not stated; it can probably be repaired, but it's the central equation. The TC0 / regular-language / S5 expressivity claims are imported from RWKV-7 by citation, not derived or tested here, so they shouldn't be in the contributions.\n\nThe reader's report is fair. I don't see a load-bearing flaw in the cross-attention mechanism itself—there isn't enough implementation detail to verify it, but nothing about the design is obviously wrong. The load-bearing flaw is empirical: the abstract and intro assert completed results that Section 4 explicitly labels preliminary. That's an internal contradiction, and it is disqualifying for this version.\n\nWho is this for? Someone tracking the RWKV/linear-attention diffusion program might skim it for the CrossWKV formulation, but it's not a paper to build on yet. I would not send it to peer review in this state. I'd desk-reject with an invitation to resubmit once the evaluation is actually complete, the FID/CLIP numbers are final with error bars, and the CrossWKV code and training/eval scripts are released.","headline":"A plausible CrossWKV design is buried under an abstract that claims completed results while Section 4 calls the evaluation preliminary and the released code is only the generic RWKV-7 layer.","tokens_in":10698,"tokens_out":2105,"would_cite":false,"duration_ms":19278,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CrossWKV claims that RWKV-7 can match DiT-class text-to-image quality with linear cost and constant memory.","keywords":["CrossWKV","RWKV-7","cross-attention","text-to-image generation","linear-complexity attention","diffusion model","weighted key-value mechanism","state-space model"],"falsifier":"Rerun the ImageNet 256x256 text-to-image benchmark to completion with the same DDPM sampling steps, CLIP ViT-L/14 scoring, and prompt sets used for DiT and Diffusion-RWKV, and compare FID and CLIP on the same sample count; if FID does not land near the reported 2.88 and CLIP near 0.33 under the matched protocol, the central quality claim fails.","tokens_in":9483,"feed_emoji":"🖼️","tokens_out":9595,"duration_ms":75394,"temperature":0.7,"pith_summary":"The paper proposes CrossWKV, a cross-attention mechanism built on RWKV-7's Weighted Key-Value (WKV) recurrence, and claims that a diffusion model using it (DIR-7) produces text-to-image quality comparable to Transformer-based models such as DiT while keeping linear computational cost and constant memory. Concretely, it reports a Fréchet Inception Distance (FID) of 2.88 and a CLIP score of 0.33 on ImageNet 256x256, alongside linear scaling in prompt length. The authors argue that because the recurrence uses a non-diagonal, input-dependent transition matrix, CrossWKV is more expressive than Transformer-style attention, citing capabilities such as regular-language recognition and S5 permutation tracking. The paper also states that its evaluation is still in progress and that the reported numbers are preliminary. If the results hold, a recurrent architecture could match Transformer quality in text-to-image generation without quadratic attention cost.","feed_headline":"A recurrent cross-attention hits 2.88 FID on ImageNet 256","feed_subtitle":"CrossWKV puts RWKV-7's linear scaling and constant memory behind DiT-class image quality, if the preliminary numbers hold.","key_machinery":"The load-bearing object is the CrossWKV module, which adapts the RWKV-7 WKV recurrence into a single-pass cross-attention layer. Its state update follows a generalized delta rule, $S_t = S_{t-1}(\\mathrm{diag}(w_t) - k_t^T (a_t \\otimes k_t)) + v_t^T k_t$, with output $y_t = r_t S_t + (r_t (p \\otimes k_t)^T) v_t$; here $S_t$ is the compressed history state, $w_t$ is a vector-valued decay, $a_t$ is an in-context learning rate, and $r_t$ is the receptance. Text embeddings are projected into the receptance stream and image features into key/value streams, so the recurrence performs global cross-attention in one pass. Low-rank adaptations (rank 64 decay, 16 value, 128 gate) and group normalization stabilize the state, and the implementation runs in either chunked mode for training or fused recurrent mode for short inference sequences. The paper argues that this non-diagonal, input-dependent transition is what lifts the model beyond $\\mathrm{TC}^0$ and gives it linear complexity $O(T \\cdot N \\cdot H)$ and constant memory.","core_discovery":"The central claim is that CrossWKV lets RWKV-7's linear-complexity WKV recurrence act as a global cross-attention layer for text-to-image generation, and that the resulting DIR-7 model matches the quality of DiT-class Transformers while using less compute and memory. On ImageNet 256x256, the paper reports FID 2.88 and CLIP score 0.33 for DIR-7-H, against DiT-XL/2's FID 2.27 and Stable Diffusion's CLIP 0.35, and it reports faster inference and lower memory than DiT at 256x256 and 512x512. The same mechanism is claimed to preserve RWKV-7's expressivity: with a non-diagonal input-dependent transition, a constant number of layers can in principle represent functions beyond the $\\mathrm{TC}^0$ class, including all regular languages and $S_5$ permutation tracking. Section 4 begins by saying the evaluation is currently in progress and that preliminary results should be interpreted with caution, so the paper's own claim is that these numbers are early but promising.","pith_inferences":["The $\\mathrm{TC}^0$, regular-language, and $S_5$ claims are asserted in the abstract and introduction but no derivation or experiment for them appears in this paper; a direct test would be to run synthetic state-tracking and regular-language benchmarks and check whether CrossWKV matches the claimed expressivity with a constant number of layers.","The headline comparison to DiT assumes identical evaluation protocols, yet Section 3.5 says inference uses a 50-step denoising schedule while Section 4.2 reports following the 250-step DDPM protocol; a matched-protocol rerun would be needed before the FID comparison is apples-to-apples.","Because CrossWKV is a generic fusion layer, the same module could be lifted into other RWKV-7 applications (audio, video, or reinforcement-learning state tracking) and tested for the same linear-scaling benefit.","If the efficiency measurements are reproduced, the practical consequence is a text-to-image model that can run long prompts and higher resolutions on limited hardware, such as the Jetson Nano numbers reported in the paper, with quality close to Transformer baselines."],"forward_implications":["If the reported numbers hold, DIR-7-H (779M parameters) reaches FID 2.88 and CLIP 0.33 on ImageNet 256x256, placing a recurrent model next to DiT-XL/2 and ahead of Diffusion-RWKV-H/2 on FID.","Prompt-length scaling would stay near-linear: the paper reports inference time rising from 0.52s to 0.70s and memory from 4.5GB to 4.7GB as prompts grow from 50 to 500 tokens, where DiT grows quadratically.","Each CrossWKV component earns its place: removing the decay LoRA, learning-rate LoRA, value LoRA, or group normalization raises FID by 0.28 to 0.48 and lowers CLIP by 0.04 to 0.07 on CIFAR10.","CrossWKV's unidirectional design would keep RWKV-7's constant-memory state while beating the bidirectional Diffusion-RWKV baseline (FID 2.88 vs 2.95 on ImageNet 256x256).","A single model would keep CLIP scores of 0.31-0.34 across English, Chinese, Spanish, and French prompts, supporting multilingual text-to-image use."],"supporting_citations":[{"why":"Supplies the DiT baseline (FID 2.27, IS 278.24) that DIR-7 claims to approach.","marker":"[Peebles and Xie, 2023]"},{"why":"Defines RWKV-7's WKV recurrence and generalized delta rule, the base architecture CrossWKV extends.","marker":"[Peng et al., 2025]"},{"why":"Provides the Diffusion-RWKV baseline, the 250-step DDPM evaluation protocol, and model-size configuration.","marker":"[Fei et al., 2024]"},{"why":"Provides the Stable Diffusion baseline and the CLIP-score comparison point of 0.35.","marker":"[Rombach et al., 2022]"},{"why":"Supplies the DDPM diffusion objective and sampling framework used for training and evaluation.","marker":"[Ho et al., 2020]"},{"why":"Supplies the CLIP text encoder and the ViT-L/14 scorer used for text conditioning and alignment scores.","marker":"[Radford et al., 2021]"},{"why":"Baseline CrossMamba and the efficient cross-modal design CrossWKV is inspired by.","marker":"[Wu et al., 2025]"}],"fun_headline_variants":["CrossWKV: RWKV-7's linear-time cross-attention for image gen","RWKV-7 cross-attention matches DiT at 2.88 FID on ImageNet","Constant-memory cross-attention: RWKV-7 hits DiT-class FID","Beyond TC0: RWKV-7's cross-attention for text-to-image","CrossWKV: Linear-scaling cross-modal attention for RWKV-7"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the reported FID 2.88 and CLIP 0.33 come from a completed, standardized evaluation whose protocol matches the baselines; the paper itself says the evaluation is currently in progress, so if the runs were partial or used a different sampling schedule, the claimed match to state-of-the-art quality is not established.","fun_headline_variants_meta":{"raw":{"variants":["CrossWKV: RWKV-7's linear-time cross-attention for image gen","RWKV-7 cross-attention matches DiT at 2.88 FID on ImageNet","Constant-memory cross-attention: RWKV-7 hits DiT-class FID","Beyond TC0: RWKV-7's cross-attention for text-to-image","CrossWKV: Linear-scaling cross-modal attention for RWKV-7"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2933,"prompt_tokens":1039,"completion_tokens":1894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":1783}},"tokens_in":655,"tokens_out":1894,"duration_ms":13345,"temperature":1.0,"reasoning_tokens":1783,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:53:33.716421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the ImageNet 256x256 text-to-image benchmark to completion with the same DDPM sampling steps, CLIP ViT-L/14 scoring, and prompt sets used for DiT and Diffusion-RWKV, and compare FID and CLIP on the same sample count; if FID does not land near the reported 2.88 and CLIP near 0.33 under the matched protocol, the central quality claim fails.","supporting_citations":[{"cited_title":"Learning transferable vi- sual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP text encoder and the ViT-L/14 scorer used for text conditioning and alignment scores."},{"cited_title":"Scalable di ffusion models with transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the DiT baseline (FID 2.27, IS 278.24) that DIR-7 claims to approach."},{"cited_title":"High-resolution image synthesis with latent di ffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the Stable Diffusion baseline and the CLIP-score comparison point of 0.35."}],"review_version":1}