{"id":"e7119c65-3045-4668-94f9-dc2f16f51db5","arxiv_id":"2504.19191","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"WuNeng combines attention heads with RWKV-7 state-driven heads and cross-head gated fusion, but the paper is an incomplete proposal with preliminary, uncontrolled results.","lead":"This preprint proposes WuNeng, a hybrid language model architecture that mixes standard attention heads with RWKV-7 recurrent-state heads. The authors report preliminary benchmarks claiming 10% to 15% gains over Qwen2.5-7B, but the evaluation is unfinished and many design details are left unspecified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 10–15% gains cannot be attributed to the WuNeng architecture because the only comparison pits the architecture plus ARWKV distillation/SFT/DPO against untouched baselines; the paper itself says evaluation is ongoing.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing gap: the reported gains are attributed to the WuNeng architecture without separating the effect of the ARWKV distillation and Stage 3 SFT/DPO pipeline from the architecture itself. This is not a minor methodological quibble. The paper's central claim is that WuNeng-7B outperforms Qwen2.5-7B-Instruct by 10–15% because of hybrid attention and cross-head mechanisms. Since WuNeng-7B is itself derived from Qwen2.5-7B-Instruct and then receives additional SFT/DPO training, a same-pipeline continued-training control on the baseline is the minimal experiment required to attribute the gap to the architecture. The paper explicitly labels the evaluation as ongoing and preliminary, and the absence of protocol details, code, and ablations makes the result unverifiable as reported. I therefore agree with the reader's REJECT verdict: the manuscript does not yet support its central empirical claim. Secondary issues such as the dimensional consistency of Eq. 11 and the undefined kernel combine F are additional signs that the manuscript is incomplete, but the decisive concern is the lack of a controlled comparison.","tokens_in":9004,"tokens_out":3770,"duration_ms":38448,"concrete_test":"Run the same Stage 3 SFT+DPO recipe described in Sections 4.1 and 4.4 — the same 770M training tokens, same 8K context length, same DPO data, and same number of steps — as continued training on Qwen2.5-7B-Instruct, then evaluate both that control and WuNeng-7B with the same harness and settings. If the control matches or exceeds WuNeng-7B's MMLU (80.33%) and GSM8K (92.22%) scores, the architectural claim is unsupported; if the control falls short by a comparable margin, the attribution concern is resolved in WuNeng's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim that WuNeng-7B is roughly 10–15% better than Qwen2.5-7B-Instruct because of hybrid attention, cross-head interactions, and multi-token state processing, the experiments must hold the training and downstream protocol fixed. They do not. Section 4.1 states that WuNeng models were distilled from Qwen2.5-7B-Instruct, and Section 4.4 says Stage 3 applied SFT and DPO on 770M tokens with an 8K context. Qwen2.5-7B-Instruct is then evaluated as an off-the-shelf checkpoint, so Table 1 conflates at least three differences: the architecture, continued supervised/preference training, and the evaluation protocol itself. The paper's own text flags the incompleteness: Section 4 opens with 'As the WuNeng architecture is an ongoing work, its evaluation is currently in progress,' and the Table 1 footnote repeats that evaluation is ongoing. No code, no training details, no eval harness version, and no ablations isolating cross-head or multi-token processing are provided. Section 4.2 still contains an unresolved citation 'as in ARWKV [?]', and Section 5 refers to 'Figure ??'. Because the headline result is an empirical claim about an architecture, and the only comparison omits a same-pipeline control, the missing control for the ARWKV/SFT/DPO pipeline is the load-bearing gap: without it, the 10–15% gap may simply reflect additional training, not WuNeng's design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces WuNeng, a hybrid large-language-model architecture that augments standard multi-head attention with RWKV-7 state-driven heads, adds cross-head interaction mechanisms (additive modulation, gated fusion, and a 'middle head' bridge), and conditions queries on a multi-token RWKV-7 state. The authors describe a three-stage ARWKV-based training pipeline (attention alignment, knowledge distillation, and SFT/DPO) and report preliminary benchmark numbers claiming WuNeng-7B outperforms Qwen2.5-7B-Instruct by 10-15% on MMLU, GSM8K, GPQA, and other tasks (Table 1). The paper explicitly states several times that evaluation is ongoing and contains unresolved placeholders, including an unresolved citation and a missing figure reference.","tokens_in":9413,"tokens_out":5346,"duration_ms":54033,"significance":"The architectural idea of keeping full attention while adding RWKV-7 state heads and explicit cross-head interactions is a reasonable direction to explore, and the paper correctly identifies that prior hybrid designs tend to prioritize efficiency over expressivity. If the reported gains were rigorously attributed to the architecture, this would be a useful contribution to the ongoing attention-versus-state-model discussion. However, the current manuscript does not support that attribution: the central comparison is confounded by unequal training, the architecture is under-specified to the point of non-reproducibility, and no ablation or proof substantiates the individual mechanisms. As submitted, the paper is closer to an extended abstract or position statement than a complete technical report.","major_comments":[{"comment":"The headline claim of a 10-15% improvement over Qwen2.5-7B-Instruct is uninterpretable because WuNeng-7B was distilled from Qwen2.5-7B-Instruct and then trained with SFT and DPO on 770M tokens (Sec 4.1, 4.4), while the baselines (Qwen2.5-7B-Instruct, Hymba-1.5B, LLaMA3.2-3B) were evaluated off-the-shelf. There is no same-pipeline control (e.g., Qwen2.5-7B-Instruct trained on the same Stage 3 data and protocol). The text's statement that baselines were evaluated 'under identical conditions' refers only to inference settings, not to training, so the measured differences cannot be attributed to the WuNeng architecture rather than to the additional training itself. This missing control is load-bearing for the paper's central claim.","section":"Sec 4.1 / Table 1"},{"comment":"The architecture is insufficiently specified for reproduction. The operator F in Eq (2) is described only as a 'kernel combine mechanism, such as concatenation, summation, or a learned transformation,' but the paper never states which variant is used in the experiments or how the combination is computed when F takes multiple argument sets. Eq (6) uses F with three argument sets (including {M_h}), whereas Eq (2) uses two, and no rule is given for reducing these to the intermediate representation W_l_attn consumes. Eq (11) adds λ W_state^h S_t to X W_Q^h; since S_t is a d×d state matrix and Q_h is a length×d query matrix, the addition is not dimensionally coherent unless W_state^h maps S_t (or a function of it) to a sequence-length representation in a way that is not described. These gaps prevent the reader from connecting the experimental results to the proposed mechanism.","section":"Sec 3.1-3.3, Eqs (2), (6), (11)"},{"comment":"The statement that the RWKV-7 state enables 'complex state manipulations such as swapping entries, which enhances expressivity beyond the TC0 complexity class' is asserted without proof or a supporting citation. If this is a formal complexity claim, a proof sketch or a precise reduction must be supplied; if it is intended to echo results from RWKV-7 or other work, the relevant reference and precise statement are needed. As written, the sentence is a bare assertion in a section that otherwise defines the architecture.","section":"Sec 3.1"},{"comment":"The convergence evidence is reported as single loss values without error bars, repeated runs, or a same-size RWKV-7-7B baseline. The text claims WuNeng-7B's hybrid attention 'converged to a loss of 0.15' in Stage 1 and 0.08 in Stage 2, but these numbers are not accompanied by variance or the number of seeds. Figure 3's caption says WuNeng-7B (green) and Qwen2.5-7B-Instruct (blue) are compared 'during knowledge distillation,' yet Qwen2.5 is a Transformer, not a RWKV-based model, making it unclear what the comparison is intended to show. These points must be clarified before the claimed convergence advantage can be assessed.","section":"Sec 4.2, 4.3, Figure 3"},{"comment":"No ablation isolates the three proposed components (hybrid heads, cross-head interactions, multi-token state processing). The only reported comparison is the full WuNeng-7B (with cross-head interactions and active MLPs, plus SFT/DPO) against untrained baselines. Consequently, even if the same-pipeline control were added, the individual contributions claimed in Sections 3.2 and 3.3 would remain unsupported; the paper needs ablations removing each mechanism to allow attribution.","section":"Sec 3.2-3.3 / Table 1"}],"minor_comments":[{"comment":"The citation 'as in ARWKV [?]' is unresolved; a proper author/year citation must be provided.","section":"Sec 4.2"},{"comment":"The text refers to 'Figure ??' for the alignment loss curves; the figure number is missing and must be fixed.","section":"Sec 5"},{"comment":"The abstract claims WuNeng 'sets a new standard for balancing expressivity and computational efficiency'; given the explicitly preliminary evaluation and the unresolved comparison issues, this overclaims and should be tempered.","section":"Abstract"},{"comment":"The notation for state-derived keys is inconsistent: the text introduces \\\\hat{K}_h and also calls it 'state-derived key,' but does not relate it to the previously defined K_h; consistent notation would improve reproducibility.","section":"Sec 3.1"},{"comment":"The caption lists the colors but the figure itself should include a legend and explicit axis labels with units; the y-axis (loss) and x-axis (steps) are never defined in the text.","section":"Figure 3"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for full peer review. It explicitly states that evaluation is ongoing, contains unresolved citation and figure placeholders, and its main empirical claim rests on a confounded comparison without a same-pipeline baseline. The architectural idea may have merit, but the authors need substantial additional work: precise formal definitions, ablations, and a controlled baseline. This is more suitable for a workshop submission in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: WuNeng is a plausible architectural sketch, but the paper's headline result doesn't survive contact with the experimental design. The 10-15% gain over Qwen2.5-7B-Instruct is confounded because WuNeng-7B gets the full ARWKV distillation/SFT/DPO pipeline while the baseline is evaluated off-the-shelf. The paper itself admits the evaluation is ongoing, so this is a status report, not a finished claim.\n\nWhat's genuinely worth a look: adding RWKV-7 state-driven heads to standard attention instead of replacing heads (Hymba's move) is a reasonable idea for boosting expressivity. The middle-head cross-attention mechanism is a variation on gated fusion and query-state modulation that already exist, but the specific combination is a stated direction. The authors cite Hymba, RWKV-7, GLA, and ARWKV, and they are upfront about the preliminary nature. That transparency is real credit.\n\nThe soft spots are load-bearing, and the stress-test note is right on target. There's no control that isolates the architecture from the training recipe. WuNeng gets 770M tokens of SFT and DPO; Qwen2.5 is evaluated as shipped. That alone can explain most of the reported gap. On the architecture side, F is defined as 'concatenation, summation, or a learned transformation' — that's a list of options, not a definition. Equation 11 adds a state vector to queries without checking dimensions. The TC0 claim is asserted without proof. The broken references ('[?]' and 'Figure ??') are cosmetic, but they add to the sense of an unfinished manuscript.\n\nWho should read this? People tracking hybrid-state/attention designs might skim it for the design direction. Nobody can reproduce the results from this text.\n\nRecommendation: not ready for referees. I'd desk-reject or ask the authors to return when the evaluation is complete, with a same-pipeline baseline, ablations for cross-head and multi-token components, and a fully specified update rule. The architectural idea may be worth revisiting then.","headline":"Incomplete draft whose headline 10-15% gain is confounded by the training pipeline; the architectural sketch is plausible but under-specified.","tokens_in":9910,"tokens_out":3718,"would_cite":false,"duration_ms":36086,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims WuNeng, a hybrid of standard attention and RWKV-7 state heads with cross-head interactions and multi-token state processing, outperforms Qwen2.5-7B-Instruct by roughly 10–15% in preliminary benchmarks.","keywords":["hybrid attention","RWKV-7","state-driven heads","cross-head interaction","multi-token state processing","language model architecture","knowledge distillation","benchmark evaluation"],"falsifier":"Run the identical supervised fine-tuning and preference-optimization recipe on Qwen2.5-7B-Instruct and compare against WuNeng-7B under the same protocol; if the gap collapses, the architectural claim is unsupported. A second check is to ablate the state-injected query term by fixing $\\lambda=0$ and measuring whether the MMLU and GSM8K margins survive.","tokens_in":8794,"feed_emoji":"🧠","tokens_out":8697,"duration_ms":70709,"temperature":0.7,"pith_summary":"The paper proposes WuNeng, an LLM architecture that keeps standard multi-head attention and adds extra heads driven by the continuous recurrent state of RWKV-7, rather than replacing attention heads for efficiency. It claims these added state-driven heads, a cross-head interaction scheme, and a multi-token state-processing trick improve contextual coherence and reasoning at a cost of under 5% extra parameters. The reported evidence is a preliminary evaluation in which WuNeng-7B scores 80.33% on MMLU and 92.22% on GSM8K, roughly 10–15% above Qwen2.5-7B-Instruct's 71.72% and 82.34%. The paper argues that expressivity, not KV-cache reduction, is the main payoff of hybrid attention-state designs.","feed_headline":"Adding RWKV-7 state heads to attention beats a 7B baseline by 10-15%","feed_subtitle":"WuNeng keeps transformer heads, adds recurrent state heads, and reports wins on MMLU, GSM8K, and GPQA.","key_machinery":"The central object is the hybrid-head layer. Standard attention $A_h = \\operatorname{softmax}(Q_hK_h^T/\\sqrt{d_k})V_h$ is augmented with state-derived keys $\\hat K_h = W_{\\hat K,h}S_tX$, and middle heads $M_h = \\sigma(W_h^{\\mathrm{mid}}(A_h + \\beta S_t^T\\hat K_h))$ bridge attention and state through learnable scalars and gates. The RWKV-7 state $S_t$ is updated by a generalized delta rule, and multi-token state processing injects the state into queries via $Q_h = XW_h^Q + \\lambda W_h^{\\mathrm{state}}S_t$. This machinery is what the paper says carries the benchmark gains, since it is the architectural addition beyond the distillation pipeline.","core_discovery":"On its own terms, the paper claims that a hybrid-head layer can outperform a strong dense-transformer baseline by a wide margin without scaling parameters. Each layer augments standard multi-head attention with RWKV-7 state-driven heads, links the two through learnable middle heads using concatenation, additive modulation, or gated fusion, and feeds the recurrent state back into the query projection as $Q_h = XW_h^Q + \\lambda W_h^{\\mathrm{state}} S_t$. In the preliminary Stage 3 comparison, WuNeng-7B reaches 80.33% on MMLU, 92.22% on GSM8K, 55.12% on GPQA, and 82.45% on IFEval, against 71.72%, 82.34%, 49.0%, and 73.62% for Qwen2.5-7B-Instruct. The paper attributes this 10–15% gap to the hybrid attention mechanism and cross-head interactions, while stating that evaluation is ongoing and the results are preliminary.","pith_inferences":["The main reported comparison is not pipeline-matched: WuNeng-7B received distillation, supervised fine-tuning, and preference optimization on 770M tokens, while the Qwen2.5-7B-Instruct numbers are the public baseline without that extra alignment, so part of the 10–15% gap could be training, not architecture.","A controlled ablation that masks out the state heads and cross-head gating while keeping the same three-stage pipeline would isolate how much of the gain each component contributes.","A targeted test of the state-injection term is to set the learnable scalar $\\lambda$ to zero and check whether the MMLU and GSM8K margins persist; if they do, the multi-token state injection is not the driver.","The design suggests a testable combination with existing sparse-attention models: adding state heads to a sparse-attention transformer could give both long-range state and local precision, something the paper only gestures at."],"forward_implications":["If the reported gains hold, adding recurrent state heads to existing transformer layers can improve reasoning benchmarks at the same parameter scale.","If the gains reproduce under controlled comparisons, the hybrid-head design gives a practical path to combining linear-time state summarization with high-resolution attention recall.","If the architecture is the cause, the under-5% parameter overhead means the approach could be applied to stronger base models without retraining from scratch.","If the mechanism works as described, multi-token state injection should also help long-context and state-tracking tasks, which the paper lists as future directions."],"supporting_citations":[{"why":"Defines the RWKV-7 state and the generalized delta rule that WuNeng's state-driven heads inherit.","marker":"[Peng et al., 2025]"},{"why":"Supplies the hybrid-head idea of mixing attention heads with state-driven heads.","marker":"[Dong et al., 2024]"},{"why":"Provides the three-stage distillation, supervised fine-tuning, and preference-optimization pipeline used to train and evaluate WuNeng.","marker":"[Yueyu et al., 2025]"},{"why":"Qwen2.5 is the teacher model and the main baseline whose benchmark numbers the 10–15% claim is measured against.","marker":"[Yang et al., 2024a]"},{"why":"LLaMA3.2-3B serves as a baseline for the Stage 3 comparison, showing how smaller models trail.","marker":"[Grattafiori et al., 2024]"},{"why":"Motivates the multi-token processing idea that WuNeng's multi-token state mechanism draws on.","marker":"[Gloeckle et al., 2024]"},{"why":"Provides the multi-token attention concept cited as inspiration for the state-based multi-token processing.","marker":"[Golovneva et al., 2025]"}],"fun_headline_variants":["RWKV-7 state heads boost attention: 10-15% gains on 7B","WuNeng: Hybrid heads beat dense transformers at same scale","Cross-head fusion with RWKV-7 states lifts 7B LLM by 10-15%","WuNeng adds recurrent heads to attention, not replacing them","Hybrid heads: RWKV-7 state plus attention outperforms by 10-15%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported 10–15% benchmark gap is caused by the WuNeng architecture itself, not by the extra distillation, supervised fine-tuning, and preference-optimization training that the baseline models did not receive.","fun_headline_variants_meta":{"raw":{"variants":["RWKV-7 state heads boost attention: 10-15% gains on 7B","WuNeng: Hybrid heads beat dense transformers at same scale","Cross-head fusion with RWKV-7 states lifts 7B LLM by 10-15%","WuNeng adds recurrent heads to attention, not replacing them","Hybrid heads: RWKV-7 state plus attention outperforms by 10-15%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3317,"prompt_tokens":948,"completion_tokens":2369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":2260}},"tokens_in":564,"tokens_out":2369,"duration_ms":15963,"temperature":1.0,"reasoning_tokens":2260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:59:08.170334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical supervised fine-tuning and preference-optimization recipe on Qwen2.5-7B-Instruct and compare against WuNeng-7B under the same protocol; if the gap collapses, the architectural claim is unsupported. A second check is to ablate the state-injected query term by fixing $\\lambda=0$ and measuring whether the MMLU and GSM8K margins survive.","supporting_citations":[{"cited_title":"ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer","cited_arxiv_id":"2501.15570","evidence_quote":"Provides the three-stage distillation, supervised fine-tuning, and preference-optimization pipeline used to train and evaluate WuNeng."}],"review_version":1}