{"id":"fc4b5f38-51e7-4b16-bb01-d356ce8f1aa8","arxiv_id":"2607.18794","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Linear-attention policy backbones with structured, weighted state updates outperform Transformer and RNN baselines in open-vocabulary object-goal navigation, with the WSLA variant giving the largest gains.","lead":"This paper shows that linear-attention state updates, instead of Transformer self-attention over a context window, improve open-vocabulary object-goal navigation, and that adding expanded per-head sub-states with learned weighting (WSLA) lifts success rate to 36.4% on HM3D-OVON. It also reports 82% success over 50 trials on a Unitree Go2 robot, evidence that the approach transfers to hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-recipe confound: context length and rollout length are varied together, so the Transformer's scaling failure in Fig. 2 may be an optimization artifact rather than a property of self-attention.","rationale":"The paper's central assertion is that the policy backbone's state-update mechanism, not context length per se, drives navigation performance. The evidence rests on Table 1 and Fig. 2. The reader's weakest assumption identifies the main risk: the DAgRL training recipe—DAgger pre-training, PPO fine-tuning, Adam lr 2.5e-4, rollout length matched to context length—may favor LA backbones and disadvantage Transformers. I agree. The most specific and testable form of this concern is in Fig. 2: l_train and rollout length are co-varied, so the claim that 'Transformer policies do not benefit from larger training contexts' is contaminated. Longer rollouts change the PPO update statistics (higher variance, different value targets, fewer updates per environment step) and can degrade Transformer training even if self-attention could exploit longer contexts. The lack of a separate rollout-length control makes the interpretation underdetermined. The same recipe was presumably developed for the original DAgRL recurrent policy, so the Transformer baseline may be undertuned; standard Transformer practice (lower LR, warmup, possibly different batch size) is not reported. This is not an accusation of unfairness; it is an identification of a missing control and a missing hyperparameter search. The paper does provide real strengths: three-seed standard deviations, matched encoders and policy inputs, and an ablation of WSLA components, which support the internal validity of the LA-family ordering. But the comparison against the Transformer, the paper's main 'beyond Transformers' claim, hinges on the fairness of the training recipe. If the proposed decoupled-rollout and LR sweep test shows no improvement for the Transformer, the central claim would be substantially confirmed. If it does improve, the headline would need to be narrowed from 'state-update mechanism' to 'state-update mechanism under a recurrent-friendly optimizer.' Therefore the conditional verdict stands, pending this test.","tokens_in":15080,"tokens_out":6685,"duration_ms":63563,"concrete_test":"Retrain the Transformer baseline on HM3D-OVON with rollout length decoupled from context length: set l_train=500 but rollout length=100 (or 300) under the same DAgger+PPO pipeline, and evaluate at l_test=500. If SR rises substantially relative to the rollout-length=500 run in Fig. 2, the context-length conclusion is confounded. Also run a minimal LR/warmup sweep (e.g., lr=1e-4 with 1000-step linear warmup) for the Transformer at l_train=500; if the best tuned Transformer reaches or exceeds WSLA's 36.4% average SR, the claim that state-update design—not optimization—drives the gain is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that LA state updates outperform Transformer self-attention depends on matched training being fair. In 'Training Details' (Sec. 4.1), the authors state: 'the rollout length is set to match the context length l.' In Fig. 2, they vary l_train in {100, 300, 500} and report that Transformer performance plateaus or degrades at l_train=500. But because rollout length is tied to l_train, the experiment simultaneously changes the length of trajectories used for PPO updates. Longer rollouts alter advantage estimation, value-target distribution, and update frequency, which can hurt Transformer optimization independently of context length. Furthermore, all models share Adam lr=2.5e-4, no warmup, and the DAgRL recipe; Transformers typically require lower LR and warmup. Thus the reproduced Transformer baseline (30.1% average SR) may be undertuned, and the 6.3 pp gap over WSLA may reflect optimization mismatch rather than the state-update mechanism. The HSSD and real-robot results do not resolve this because they compare only the same undertuned Transformer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LANav, a linear-attention policy backbone for Open-Vocabulary Object Goal Navigation (OVON), and its variant WSLA, which expands each attention head into multiple sub-states with learned weighted readout. The central claim is that, under matched training and architecture settings, linear-attention/state-update backbones outperform RNN and Transformer baselines on HM3D-OVON, with WSLA reaching 36.4% average SR versus 30.1% for the best Transformer, and that the advantage grows with longer training contexts. The paper also reports distance-stratified gains, HSSD zero-shot and fine-tuned transfer, and a real-robot deployment on a Unitree Go2 with 82% success over 50 trials.","tokens_in":15395,"tokens_out":2349,"duration_ms":25800,"significance":"If the central claim holds, the paper identifies a concrete and potentially general design principle for partially observable embodied policies: structured recurrent state updates are more effective than fixed-window self-attention for integrating navigation history, independent of sequence-length complexity. The study is significant in scope because it uses an external benchmark (HM3D-OVON), matches the policy backbone while fixing encoders and training pipeline, reports three-seed means and standard deviations for the main table, and includes both simulated and real-world transfer results. The parameter-count and FLOPs analysis in Fig. 4 also makes the efficiency argument concrete. However, the claim is load-bearing on the fairness of the shared training recipe, and several supporting analyses lack uncertainty quantification or a decoupled experimental design.","major_comments":[{"comment":"The central 'LA benefits from longer contexts; Transformer does not' claim is confounded. Training Details states 'the rollout length is set to match the context length l.' Figure 2 varies l_train in {100, 300, 500}, thus simultaneously varying the PPO rollout length. Longer rollouts alter advantage estimation, value-target distribution, and update frequency, which can independently affect Transformer optimization. The reported plateau/degradation of Transformer at l_train=500 may therefore be an artifact of the rollout schedule, not a property of self-attention. Please rerun Fig. 2 with rollout length fixed while varying only the context window, or otherwise decouple these two variables.","section":"Sec. 4.1 / Fig. 2"},{"comment":"All reproduced methods share Adam lr=2.5e-4, no warmup, and the DAgRL recipe. Transformers are commonly sensitive to learning rate and warmup, and the reproduced Transformer at 30.1% may be undertuned relative to the LA variants. Since the headline 6.3 pp gap between WSLA and Transformer is the paper's main evidence, a learning-rate or warmup sensitivity study for the Transformer baseline is required to show the comparison is not an optimization mismatch. At minimum, report results with a Transformer tuned under the same budget (e.g., lr sweep or a brief warmup schedule).","section":"Table 1 / Sec. 4.1"},{"comment":"Figure 2 and the HSSD results (Table 4) are reported without error bars or multiple seeds. The main table includes three-seed means, but the context-length and HSSD claims are equally central. Please provide means and standard deviations over at least three seeds for these analyses, or clearly state if only one seed was used and justify why the observed differences are significant.","section":"Fig. 2 / Table 4"},{"comment":"The paper positions LA as an alternative to both RNN and Transformer, and argues that structured state update is the key mechanism. However, no selective state-space model (e.g., S4, Mamba, or a gated SSM baseline) is included in the comparisons. Such models also maintain a structured recurrent state and are widely used in decision-making. Without this baseline, the claim that linear attention specifically—rather than any recurrent state-update backbone—is responsible for the gains remains incomplete. Please add at least one selective-SSM baseline under matched settings, or explicitly delimit the claim to the LA family.","section":"Method / Related Work"}],"minor_comments":[{"comment":"The abstract uses 'constant FLOPs' phrasing in the efficiency claim; Fig. 4 shows near-constant but not exactly constant online FLOPs. Suggest 'nearly constant' for precision.","section":"Abstract / Page 1"},{"comment":"The RMSNorm applied after concatenation is not described in the text; please specify whether it is applied over the feature dimension only, and whether the gate input z_t is the backbone-layer hidden state.","section":"Sec. 3.3 / Eq. (18)"},{"comment":"The right panel uses a log-scale y-axis without labeling it as log. Add a clear axis label so the reader can interpret the FLOPs comparison directly.","section":"Fig. 4"},{"comment":"The EALM row is marked N/A for VAL SEEN SYN and is described as using a different training strategy. Clarify in the table caption whether EALM uses the same visual and language encoders, since that affects comparability.","section":"Sec. 4.2 / Table 1"},{"comment":"The ablation in Table 3 reports SR increases of 34.6->35.2->36.4 without error bars. Given the main table shows seed-level variation of ~0.3, please report means and standard deviations for the ablation as well.","section":"Sec. 4.4 / Table 3"},{"comment":"The limitations sentence is brief; it mentions shallow backbones and bounded context windows but not the absence of code. Since the paper states that code will be made public, please either provide a link in the final version or explicitly state that it is available upon request.","section":"Conclusion"},{"comment":"Several references to the authors' own prior work (MetaLA, Spiking-Brain, MMDend) appear in the related work and LA family discussion. While these citations are topically relevant, please double-check that they are cited for their technical content and not simply as self-promotion; the current text appears acceptable.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question: whether linear-attention state updates are a better inductive bias than self-attention for embodied policies under partial observability. The main table is well controlled and the real-robot result is a useful plus. My concern is not with the direction but with the fairness of the comparison: the rollout-length confound in Fig. 2 and the shared hyperparameter recipe for the Transformer baseline are load-bearing. If the authors can decouple rollout length from context length and show the Transformer is not undertuned, the central claim would be substantially strengthened. I also recommend adding a selective-SSM baseline, since the paper's framing invites that comparison. With these changes I would be willing to consider acceptance; without them, the evidence is not yet strong enough for the strong claims made in the abstract and introduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the take: this is a worthwhile paper that should go to peer review, but the headline comparison has a real confound that referee time should focus on.\n\nWhat's genuinely new: the empirical result that, holding the DAgRL recipe fixed, linear-attention backbones beat a reproduced Transformer on HM3D-OVON, with the gain growing as the state update becomes more structured (Linear Transformer 31.3 → DeltaNet 33.9 → Gated DeltaNet 34.6 → WSLA 36.4). The three-seed means with stds are credible, and the efficiency analysis is honest: per-action FLOPs stay near 0.05G for LA while the Transformer's cost grows linearly with context length. The HSSD zero-shot and fine-tuned transfer and the 50-trial real-robot deployment are useful supporting evidence, even if the robot part is a single environment with no baseline.\n\nThe soft spots are exactly the stress-test ones. The biggest is Fig. 2: because the rollout length is set to match the context length, the l_train sweep changes both the context window and PPO's rollout truncation. The Transformer's plateau at l_train=500 could be an optimization artifact (longer rollouts alter advantage estimation and update frequency) rather than a property of self-attention. On top of that, all models share Adam lr=2.5e-4 with no warmup—a standard LSTM/PPO recipe that is likely suboptimal for Transformers. So the 6.3 pp WSLA advantage in Table 1 may be partly a tuning gap, not purely a state-update mechanism advantage. That doesn't invalidate the paper, but it means the central mechanistic claim needs a cleaner test: decouple rollout length from context, or tune Transformer LR/warmup separately.\n\nMinor but real: no code or configs despite the promised release; no error bars in Fig. 2, Table 2, or Table 4; no Mamba/S4 selective-SSM baseline to position the LA family; and the real-robot result is feasibility evidence, not a benchmark. The citation pattern looks fine—self-citations are to relevant related work and don't appear load-bearing. The paper is also transparent about its limitations.\n\nVerdict: send it to review, but ask for the confound analysis and code release. As it stands I'd cite it as useful evidence that LA backbones are worth trying in navigation, with a caveat about the tainted comparison; if the confound gets resolved, the main claim becomes much stronger.","headline":"A worthwhile empirical study of linear-attention backbones for OVON with credible three-seed results, but the headline comparison is partly confounded by tying rollout length to context length and an undertuned Transformer baseline.","tokens_in":15877,"tokens_out":3241,"would_cite":true,"duration_ms":31868,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that linear-attention policy backbones, which compress history into structured recurrent states, outperform RNN and Transformer backbones for open-vocabulary object goal navigation, with a proposed WSLA variant reaching 36","keywords":["open-vocabulary object goal navigation","linear attention","policy backbone","state-space models","partially observable navigation","success rate","embodied navigation","WSLA"],"falsifier":"Train the Transformer baseline with a per-backbone hyperparameter search (learning rate, rollout length, context length) or with the same state-update inductive bias; if it then matches or exceeds the 36.4% average success rate on HM3D-OVON, the claim that self-attention cannot exploit longer histories in OVON fails.","tokens_in":15003,"feed_emoji":"🤖","tokens_out":4194,"duration_ms":40789,"temperature":0.7,"pith_summary":"The paper is trying to establish that the way a navigation policy updates its internal state—not how much history it can attend to—is the key driver of performance under partial observability. It argues that self-attention-based Transformers, despite their explicit context window, do not exploit longer histories on the open-vocabulary object goal navigation (OVON) task. In contrast, linear-attention backbones, which maintain a structured recurrent state updated from each observation, consistently outperform both RNN and Transformer baselines under matched settings. The authors introduce WSLA, a linear-attention variant that expands each head into multiple regulated sub-states with learned readout weighting, achieving a 6.3 percentage-point average success-rate gain over the best Transformer. If true, this points to a concrete design direction for embodied policies: invest in state-update design rather than context-length scaling.","feed_headline":"36.4% success: linear-attention policy beats Transformer backbones","feed_subtitle":"Structured memory updates, not longer context, drive success in partially observable navigation.","key_machinery":"The central object is the linear-attention head as a recurrent associative memory: instead of retaining a window of tokens and recomputing self-attention, each head maintains a state matrix S_t that is incrementally updated from the current key-value pair and read out by the current query. WSLA extends each of the H heads into E independent sub-states, each with its own retention decay and write coefficient (a gated delta rule), and then aggregates the sub-state readouts with a learnable softmax-weighted combination shared across heads and time. This machinery replaces quadratic-window self-attention with a near-constant per-step update that compresses history into a fixed-size structured st","core_discovery":"Under a controlled comparison that fixes visual and language encoders, training pipeline, and optimization settings, the paper finds that linear-attention policy backbones outperform both RNN and Transformer backbones on HM3D-OVON. Transformer policies do not improve with longer training contexts—performance can even degrade—whereas linear-attention backbones improve with larger training context lengths and stay stable across test-time lengths. The proposed WSLA, which expands each attention head into multiple sub-states with independent gated delta updates and a learnable weighted readout, reaches a 36.4% average success rate versus 30.1% for the strongest Transformer baseline, with the lar","pith_inferences":["The same state-expansion-plus-weighted-readout recipe could transfer to other partially observable RL tasks beyond navigation—such as manipulation or search—where the policy must decide what to write into memory and when to overwrite it.","The paper's finding that Transformer gains do not scale with context length under the given training recipe suggests that matching rollout length to context window may not be the right training protocol for Transformer policies; a per-backbone schedule could narrow the gap.","Since the largest gains appear on long-distance episodes, a testable extension would be to make the state-update rule explicitly sensitive to goal-relevant cues, e.g., by learning a write gate conditioned on the semantic similarity between the current observation and the goal description.","The constant per-step cost of linear-attention backbones suggests they could extend beyond the 500-step training horizon used here, and the paper's bounded-context limitation could be relaxed by allowing the state to persist across episode resets."],"forward_implications":["Linear-attention backbones provide a third design choice beyond RNNs and Transformers for partially observable navigation policies, improving average success rate by 6.3 points over the best Transformer baseline on HM3D-OVON.","Transformer policies do not benefit from longer training contexts in OVON, whereas linear-attention backbones do, suggesting the state-update mechanism is more important than the context window for history integration.","Within the linear-attention family, more structured and regulated state-update rules (Linear Transformer → DeltaNet → Gated DeltaNet → WSLA) yield progressively better navigation performance.","WSLA transfers to the HSSD benchmark with a 9.6-point zero-shot SR gain and an 8.3-point fine-tuned SR gain over the Transformer, and it increases SPL substantially after fine-tuning, indicating more efficient adaptation.","WSLA maintains near-constant online FLOPs (about 0.048G per action) as context length grows, unlike the Transformer whose cost grows quadratically, making the approach more practical for on-device or long-horizon deployment."],"fun_headline_variants":["Linear attention beats Transformers for object goal navigation","36.4% SR: structured memory policy outdoes Transformer backbones","Longer context hurts Transformer OVON policies, linear attention wins","Policy backbone key: linear attention tops Transformers in OVON"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison assumes the imitation-and-PPO training recipe, with rollout length matched to context length, suits the Transformer as well as the linear-attention backbones; if the Transformer baseline were simply undertuned, the reported gap would shrink.","fun_headline_variants_meta":{"raw":{"variants":["Linear attention beats Transformers for object goal navigation","36.4% SR: structured memory policy outdoes Transformer backbones","Longer context hurts Transformer OVON policies, linear attention wins","Policy backbone key: linear attention tops Transformers in OVON"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2401,"prompt_tokens":820,"completion_tokens":1581,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1509}},"tokens_in":564,"tokens_out":1581,"duration_ms":11164,"temperature":1.0,"reasoning_tokens":1509,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:16:29.103699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the Transformer baseline with a per-backbone hyperparameter search (learning rate, rollout length, context length) or with the same state-update inductive bias; if it then matches or exceeds the 36.4% average success rate on HM3D-OVON, the claim that self-attention cannot exploit longer histories in OVON fails.","supporting_citations":[],"review_version":1}