Pith. sign in

REVIEW 4 major objections 5 minor 18 references

Cross-attention for State-based model RWKV-7

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CrossWKV claims that RWKV-7 can match DiT-class text-to-image quality with linear cost and constant memory.

desk verdict A plausible CrossWKV design is buried under an abstract that claims completed results while Section 4 calls the evaluation preliminary and the released code is only the generic RWKV-7 layer. read the letter →

arxiv 2504.14260 v1 pith:LFS2FRHT submitted 2025-04-19 cs.CV cs.CL

Liu Xiao , Li Zhiyuan , Lin Yueyu This is my paper · ORCID
classification cs.CVcs.CL
keywords CrossWKVRWKV-7cross-attentiontext-to-imagegenerationlinear-complexityattentiondiffusionmodelweightedkey-valuemechanismstate-space
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CrossWKV, a cross-attention mechanism built on RWKV-7's Weighted Key-Value (WKV) recurrence, and claims that a diffusion model using it (DIR-7) produces text-to-image quality comparable to Transformer-based models such as DiT while keeping linear computational cost and constant memory. Concretely, it reports a Fréchet Inception Distance (FID) of 2.88 and a CLIP score of 0.33 on ImageNet 256x256, alongside linear scaling in prompt length. The authors argue that because the recurrence uses a non-diagonal, input-dependent transition matrix, CrossWKV is more expressive than Transformer-style attention, citing capabilities such as regular-language recognition and S5 permutation tracking. The paper also states that its evaluation is still in progress and that the reported numbers are preliminary. If the results hold, a recurrent architecture could match Transformer quality in text-to-image generation without quadratic attention cost.

What carries the argument

The load-bearing object is the CrossWKV module, which adapts the RWKV-7 WKV recurrence into a single-pass cross-attention layer. Its state update follows a generalized delta rule, $S_t = S_{t-1}(\mathrm{diag}(w_t) - k_t^T (a_t \otimes k_t)) + v_t^T k_t$, with output $y_t = r_t S_t + (r_t (p \otimes k_t)^T) v_t$; here $S_t$ is the compressed history state, $w_t$ is a vector-valued decay, $a_t$ is an in-context learning rate, and $r_t$ is the receptance. Text embeddings are projected into the receptance stream and image features into key/value streams, so the recurrence performs global cross-attention in one pass. Low-rank adaptations (rank 64 decay, 16 value, 128 gate) and group normalization stabilize the state, and the implementation runs in either chunked mode for training or fused recurrent mode for short inference sequences. The paper argues that this non-diagonal, input-dependent transition is what lifts the model beyond $\mathrm{TC}^0$ and gives it linear complexity $O(T \cdot N \cdot H)$ and constant memory.

What would settle it

Rerun the ImageNet 256x256 text-to-image benchmark to completion with the same DDPM sampling steps, CLIP ViT-L/14 scoring, and prompt sets used for DiT and Diffusion-RWKV, and compare FID and CLIP on the same sample count; if FID does not land near the reported 2.88 and CLIP near 0.33 under the matched protocol, the central quality claim fails.

Watch

Extended reading notes

Core claim

The central claim is that CrossWKV lets RWKV-7's linear-complexity WKV recurrence act as a global cross-attention layer for text-to-image generation, and that the resulting DIR-7 model matches the quality of DiT-class Transformers while using less compute and memory. On ImageNet 256x256, the paper reports FID 2.88 and CLIP score 0.33 for DIR-7-H, against DiT-XL/2's FID 2.27 and Stable Diffusion's CLIP 0.35, and it reports faster inference and lower memory than DiT at 256x256 and 512x512. The same mechanism is claimed to preserve RWKV-7's expressivity: with a non-diagonal input-dependent transition, a constant number of layers can in principle represent functions beyond the $\mathrm{TC}^0$ class, including all regular languages and $S_5$ permutation tracking. Section 4 begins by saying the evaluation is currently in progress and that preliminary results should be interpreted with caution, so the paper's own claim is that these numbers are early but promising.

Load-bearing premise

The load-bearing assumption is that the reported FID 2.88 and CLIP 0.33 come from a completed, standardized evaluation whose protocol matches the baselines; the paper itself says the evaluation is currently in progress, so if the runs were partial or used a different sampling schedule, the claimed match to state-of-the-art quality is not established.

Editorial extensions

If this is right

  • If the reported numbers hold, DIR-7-H (779M parameters) reaches FID 2.88 and CLIP 0.33 on ImageNet 256x256, placing a recurrent model next to DiT-XL/2 and ahead of Diffusion-RWKV-H/2 on FID.
  • Prompt-length scaling would stay near-linear: the paper reports inference time rising from 0.52s to 0.70s and memory from 4.5GB to 4.7GB as prompts grow from 50 to 500 tokens, where DiT grows quadratically.
  • Each CrossWKV component earns its place: removing the decay LoRA, learning-rate LoRA, value LoRA, or group normalization raises FID by 0.28 to 0.48 and lowers CLIP by 0.04 to 0.07 on CIFAR10.
  • CrossWKV's unidirectional design would keep RWKV-7's constant-memory state while beating the bidirectional Diffusion-RWKV baseline (FID 2.88 vs 2.95 on ImageNet 256x256).
  • A single model would keep CLIP scores of 0.31-0.34 across English, Chinese, Spanish, and French prompts, supporting multilingual text-to-image use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $\mathrm{TC}^0$, regular-language, and $S_5$ claims are asserted in the abstract and introduction but no derivation or experiment for them appears in this paper; a direct test would be to run synthetic state-tracking and regular-language benchmarks and check whether CrossWKV matches the claimed expressivity with a constant number of layers.
  • The headline comparison to DiT assumes identical evaluation protocols, yet Section 3.5 says inference uses a 50-step denoising schedule while Section 4.2 reports following the 250-step DDPM protocol; a matched-protocol rerun would be needed before the FID comparison is apples-to-apples.
  • Because CrossWKV is a generic fusion layer, the same module could be lifted into other RWKV-7 applications (audio, video, or reinforcement-learning state tracking) and tested for the same linear-scaling benefit.
  • If the efficiency measurements are reproduced, the practical consequence is a text-to-image model that can run long prompts and higher resolutions on limited hardware, such as the Jetson Nano numbers reported in the paper, with quality close to Transformer baselines.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CrossWKV, a cross-attention mechanism for the RWKV-7 state-based recurrent architecture, and integrates it into a text-to-image diffusion pipeline called DIR-7. It claims that CrossWKV performs global cross-modal fusion in a single pass with linear complexity and constant memory, and that DIR-7 achieves an FID of 2.88 and CLIP score of 0.33 on ImageNet 256x256, matching DiT-class models. The paper also claims that the non-diagonal input-dependent transition matrix of RWKV-7 enables representing functions beyond TC^0, including all regular languages and S5 permutation tracking. The authors state that the evaluation is in progress and provide a link to a generic RWKV-7 layer implementation.

Significance. If the empirical claims were substantiated, the paper could be significant: it proposes a plausible way to extend a linear-complexity recurrent architecture to cross-modal text-to-image generation, with potential efficiency advantages over transformer-based cross-attention. The connection to RWKV-7's expressivity is interesting, though it is not elaborated here. The paper is also transparent in stating that the evaluation is preliminary, which is commendable. However, the central empirical claim that DIR-7 matches state-of-the-art FID/CLIP scores is not supported by the submitted manuscript: the authors explicitly label the results as in-progress, no cross-attention code is provided, and the evaluation protocol has unresolved inconsistencies. The expressivity claims are imported from the RWKV-7 paper without any derivation or experiment specific to CrossWKV, so they do not add independent support.

major comments (4)
  1. [§4 and Abstract] The abstract and Table 1 present FID 2.88 and CLIP 0.33 as achieved results, but the opening paragraph of Section 4 states that the evaluation 'is currently in progress' and that the results are 'preliminary' and 'subject to further validation.' This is a direct contradiction of the paper's central claim. The empirical headline is load-bearing for the contribution, and the manuscript itself provides the grounds for not treating these numbers as established results. The code link in Section 4 points to https://github.com/TorchRWKV/flash-linear-attention/blob/dev/fla/layers/rwkv7.py, which is a generic RWKV-7 layer, not an implementation of CrossWKV or the DIR-7 training/evaluation pipeline; this makes the reported results unreproducible from the supplied artifact.
  2. [§3.1, Eq. (1)] Equation (1) is dimensionally inconsistent as printed. With S_t ∈ R^{N×N}, diag(w_t) ∈ R^{N×N}, and k_t, v_t ∈ R^N as defined, the term v_t^T k_t is a scalar, so the recurrence adds a scalar to an N×N matrix. If v_t^T k_t is intended to be an outer product or a rank-1 matrix, the notation must be changed (e.g., v_t k_t^T). A similar issue appears in Eq. (3), where k_i^T v_i is treated as a scalar while surrounded by matrix quantities. Because this recurrence is the core of the proposed mechanism, the missing convention makes the method ambiguous.
  3. [§1 and §4.8] The abstract and Introduction state that CrossWKV can represent functions beyond TC^0, including all regular languages and S5 permutation tracking, and that it performs 'board game modeling' such as Othello strategy optimization. No derivation, theorem, or experiment in this paper supports these claims; they are attributes of RWKV-7 from the cited work [Peng et al., 2025]. It is not shown that the cross-attention adaptation preserves the expressivity of the base architecture, nor is S5 or regular-language behavior tested for CrossWKV. These claims are part of the paper's positioning, so they need at least an argument or a reference to a specific result that directly covers the proposed modification.
  4. [§3.5, §4.1, §4.2] The inference and evaluation protocols are inconsistent. Section 3.5 and Section 4.1 state that inference uses a 50-step denoising schedule, while Section 4.2 says that FID is computed following Diffusion-RWKV's protocol with 250 DDPM steps. The reader cannot tell which schedule produced the numbers in Table 1, or whether the 50-step schedule is a separate experiment. Additionally, Table 1 reports no standard deviation or error bars, and Section 4.2 states that only 10K samples were generated, which is small for ImageNet FID at 256x256; these concerns compound the preliminary-status problem.
minor comments (5)
  1. [§1] The last paragraph of the Introduction refers to 'Section ??' for limitations and future work; this reference is unresolved.
  2. [§2.1] The related work cites Stable Diffusion as [Blattmann et al., 2023], which is a stable video diffusion paper, while the actual Stable Diffusion paper is [Rombach et al., 2022]. The reference should be corrected.
  3. [§4.5] Table 2 is not referenced in the text; the ablation description would benefit from an explicit pointer to the table.
  4. [Throughout] The manuscript alternates between 'DIR-7' and 'CrossWKV' when describing the model being evaluated; the relationship between the module and the full system should be stated more clearly at each use.
  5. [Abstract] The code URL in the abstract points to the generic flash-linear-attention repository, not to a repository containing the CrossWKV implementation or the DIR-7 training/evaluation code; this should be clarified or corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline FID/CLIP numbers are external measurements, and the expressivity claims are imported from RWKV-7 rather than derived, which is a support gap but not a circular reduction.

full rationale

The paper's central empirical claim—FID 2.88 and CLIP 0.33 on ImageNet 256x256—is a measurement against external benchmarks (FID, CLIP, Inception Score), not a quantity derived from the model's own assumptions. Section 4 explicitly labels the evaluation as 'currently in progress' and the results as preliminary; that is a validity/reproducibility concern, not circularity: the reported metrics are not fitted inputs renamed as predictions. The expressivity claims (TC0, regular languages, S5 tracking) are inherited by citation from Peng et al. 2025 (RWKV-7). The present authors do not overlap with that work, so this is not a self-citation, and CrossWKV's use of the same WKV equations makes the inheritance a straightforward transfer rather than a definitional restatement. The equations in Section 3.1 are presented as the RWKV-7 mechanism, not derived here, and no step in the paper reduces a predicted quantity to the data used to fit it. The unavailable CrossWKV implementation and dimensional ambiguities in Eq. (1) undermine reproducibility but do not constitute circular reasoning. Therefore no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central result rests on imported expressivity results from RWKV-7, an unproven linear-attention approximation, and the assumption that padded text alignment and the reported evaluation are valid. Free parameters include a fixed decay constant, manually chosen LoRA ranks and head geometry, and a fixed training budget. No new physical entities are introduced; CrossWKV is listed as an invented component with no independent evidence.

free parameters (3)
  • Fixed decay scaling constant c_w = -0.6065306597126334
    Appears in Eq. (8) multiplying sigmoid(LoRA(x_w)); the value is stated without derivation or fitting rationale, yet it sets the base decay scale of the WKV state update on which CrossWKV depends.
  • LoRA ranks and head geometry (decay 64, lr 64, value 16, gate 128; H=16, N=64, state 512)
    Chosen by hand in Section 3.5 and confirmed by ablation in Section 4.5; the reported FID/CLIP depend on these choices.
  • Training budget (learning rate 1e-4, batch size 256, 100K iterations on 8 A100s, 50-step inference)
    Fixed schedule stated in Sections 3.4 and 4.1 with no sensitivity analysis; adequacy for the 779M-parameter model is assumed.
assumptions (4)
  • domain assumption RWKV-7's generalized delta rule is more expressive than TC0 and can model all regular languages and S5 permutation tracking, as claimed in Peng et al. 2025.
    Invoked in Section 1 and Section 3.1 as background, but no derivation or experiment for CrossWKV is provided; the claim is imported from the cited RWKV-7 preprint.
  • domain assumption The linear-attention expansion in Eq. (3) is a valid approximation of the state update in Eq. (1) and preserves the cross-modal alignment needed for image generation.
    Equation (3) is stated as an approximation without proof; the whole linear-time argument depends on it. Section 3.1.
  • ad hoc to paper Padding text embeddings to image sequence length T and projecting them to receptance r preserves semantic alignment across modalities.
    Equation (6) pads the text with zeros and Eq. (7) feeds it to a linear layer; the paper asserts alignment but gives no analysis. Section 3.2.
  • ad hoc to paper The Table 1 metrics can be treated as final results despite the Section 4 statement that evaluation is in progress.
    The abstract and Table 1 rely on completed comparisons, while Section 4's opening paragraph explicitly cautions against relying on the numbers. This assumption is required to accept the central claim.
invented entities (1)
  • CrossWKV cross-attention layer
    purpose: Fuses text and image modalities through a single RWKV-7 WKV state update for diffusion conditioning.
    It is a proposed architectural component, not a physical entity, and its only evidence is the self-reported in-progress evaluation; no independent code, checkpoints, or falsifiable predictions outside the paper are supplied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-attention for State-based model RWKV-7." pith.science (2026). https://pith.science/paper/LFS2FRHT

@misc{pith2026250414260,
  author       = {Pith},
  title        = {Pith review of: Cross-attention for State-based model RWKV-7},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LFS2FRHT}},
  note         = {Machine review of arXiv:2504.14260}
}
abstract

We introduce CrossWKV, a novel cross-attention mechanism for the state-based RWKV-7 model, designed to enhance the expressive power of text-to-image generation. Leveraging RWKV-7's linear-complexity Weighted Key-Value (WKV) architecture, CrossWKV integrates text and image modalities in a single pass, utilizing a generalized delta rule with vector-valued gating and low-rank adaptations (LoRA) to achieve superior cross-modal alignment. Unlike Transformer-based models, CrossWKV's non-diagonal, input-dependent transition matrix enables it to represent complex functions beyond the $\mathrm{TC}^0$ complexity class, including all regular languages, as demonstrated by its ability to perform state-tracking tasks like $S_5$ permutation modeling. Evaluated within the Diffusion in RWKV-7 (DIR-7) on datasets such as LAION-5B and ImageNet, CrossWKV achieves a Frechet Inception Distance (FID) of 2.88 and a CLIP score of 0.33 on ImageNet 256x256, matching state-of-the-art performance while offering robust generalization across diverse prompts. The model's enhanced expressivity, combined with constant memory usage and linear scaling, positions it as a powerful solution for advanced cross-modal tasks, with potential applications in high-resolution generation and dynamic state manipulation.Code at https://github.com/TorchRWKV/flash-linear-attention

Figures

Figures reproduced from arXiv: 2504.14260 by the authors.

Figure 1
Figure 1. the CrossWKV mechanism global context, enhancing text-image alignment. By in￾tegrating low-rank adaptations and fused operations, DIR-7 balances efficiency and quality, making it a prac￾tical solution for scalable text-to-image generation. 3 Methodology We propose DIR-7, a text-to-image generation model that integrates the RWKV-7 architecture [Peng et al., 2025] with a novel CrossWKV module to perform global cross-a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages

  1. [1]

    Stable video diffusion: Scaling latent video di ffusion models to large datasets

    [Blattmann et al., 2023] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion En- glish, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video di ffusion models to large datasets. arXiv preprint arXiv:2311.15127 ,

    arXiv 2023

  2. [8]

    Denoising di ffusion probabilistic models

    [Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising di ffusion probabilistic models. Advances in neural information processing systems , 33:6840–6851,

    work page 2020

  3. [11]

    Scalable di ffusion models with transformers

    [Peebles and Xie, 2023] William Peebles and Saining Xie. Scalable di ffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205,

    work page 2023

  4. [12]

    Rwkv: Reinventing rnns for the transformer era,

    [Peng et al., 2023] Bo Peng, Bo Li, Wenhan Dai, Shu- jian Zhang, Jianzhong Qi, Wenjun Zeng, and Xuewei Li. Rwkv: Reinventing rnns for the transformer era,

    2023

  5. [13]

    Rwkv-7” goose” with ex- pressive dynamic state evolution

    [Peng et al., 2025] Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Haowen Hou, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, et al. Rwkv-7” goose” with ex- pressive dynamic state evolution. arXiv preprint arXiv:2503.14456,

    arXiv 2025

  6. [15]

    Zero-shot text-to- image generation

    [Ramesh et al., 2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to- image generation. In International conference on ma- chine learning, pages 8821–8831. Pmlr,

    work page 2021

  7. [16]

    High-resolution image synthesis with latent di ffusion models

    [Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image synthesis with latent di ffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695,

    work page 2022

  8. [18]

    Cross-attention inspired selec- tive state space models for target sound extraction

    [Wu et al., 2025] Donghang Wu, Yiwen Wang, Xihong Wu, and Tianshu Qu. Cross-attention inspired selec- tive state space models for target sound extraction. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2025

    work page 2025

Show all 18 references
  1. [2010]

    Long short-term memory

    [Graves and Graves, 2012] Alex Graves and Alex Graves. Long short-term memory. Supervised se- quence labelling with recurrent neural networks , pages 37–45,

    2012

  2. [2012]

    Mamba: Linear-time sequence modeling with selective state spaces

    [Gu and Dao, 2023] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

    2023 arXiv

  3. [2017]

    U-net: deep learning for cell counting, detection, and morphometry

    [Falk et al., 2019] Thorsten Falk, Dominic Mai, Robert Bensch, ¨Ozg¨un C ¸ ic ¸ek, Ahmed Abdulkadir, Yassine Marrakchi, Anton B ¨ohm, Jan Deubner, Zoe J ¨ackel, Katharina Seiwald, et al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1):67–70,

    2019

  4. [2019]

    Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models

    [Fei et al., 2024] Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models. arXiv preprint arXiv:2404.04478 ,

    2024 arXiv

  5. [2020]

    Perceiver io: A general architecture for structured inputs & outputs

    [Jaegle et al., 2021a] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint a...

    arXiv

  6. [2021]

    Decoupled weight decay regulariza- tion

    [Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regulariza- tion. arXiv preprint arXiv:1711.05101,

    2017 arXiv

  7. [2022]

    Denoising diffusion implicit models

    [Song et al., 2020] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502,

    2020 arXiv

  8. [2023]

    Gate-variants of gated recurrent unit (gru) neural networks

    [Dey and Salem, 2017] Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS) , pages 1597–1600. IEEE,

    2017

  9. [2024]

    Understanding the difficulty of training deep feedforward neural networks

    [Glorot and Bengio, 2010] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelli- gence and statistics, pages 249–256. JMLR Workshop and Confere...

    2010

  10. [2025]

    Learning transferable vi- sual models from natural language supervision

    [Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable vi- sual models from natural language supervision. In International confere...

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.