{"id":"90e37fba-d05d-47bd-abaf-3ee729e552ea","arxiv_id":"2505.11153","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DBGFQN swaps transformer feed-forward networks for a single BiGRU layer and reports improved average success rate on 23 POMDP environments, but the headline gains and parameter reductions are not reproducible from the reported tables.","lead":"This paper replaces the feed-forward layers inside a transformer-based reinforcement learning agent with a single bidirectional GRU layer, calling the result DBGFQN, and reports higher average success rates on 23 partially observable gridworld tasks. The idea is relevant for building compact memory models for RL in partially observable settings, but the headline margins and parameter savings are not clearly supported by the paper's own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BiGRU's backward pass may leak future observations into Q-values; the paper never establishes causal inference, and the reported margins are not reproducible from Table 1.","rationale":"Good-faith reading: the idea of replacing FFN with a GRU is plausible and some environments (e.g., Memory 7x7, Keydoor 9x9) show large gains in Table 1. But the architecture's validity as an RL policy depends on causality. Bidirectional recurrence is acausal by construction; standard practice either uses only the last output of a history-only window or applies a causal mask. The paper specifies neither. The phrase 'Q-values at each timestep' suggests a full-sequence forward pass, in which case backward-GRU outputs at early positions contain later observations. Since DTQN's attention is masked but the BiGRU is not, this is a concrete, testable defect, not a stylistic complaint. The second issue is that the headline percentages are not derivable from Table 1. For example, the ratio-of-means DBGFQN/DTQN is about 1.28 (i.e., +28%), while the mean of per-environment ratios is about 1.96 (+96%); neither is 87.39%, and a similar mismatch exists for DRQN. This undermines the strongest quantitative claim independent of causality. However, because neither issue can be resolved without code or a precise algorithmic description, the appropriate action is to require those artifacts and rerun the causal ablation, not to reject the architectural idea on the current evidence. The reader's conditional verdict is therefore appropriate.","tokens_in":11991,"tokens_out":10022,"duration_ms":88687,"concrete_test":"Obtain or reconstruct the evaluation loop from the paper. Insert a check at each decision step t: the tensor passed to `self.biGRU` must contain no observation with time index > t, and the chosen action must use only the last output position. Then rerun the 23-environment suite with a causal BiGRU (backward recursion truncated so h_{t+1}=0 for all t) and compute the DBGFQN-vs-DTQN and DBGFQN-vs-DRQN margins on seed-level means. If the margins in Section 5.3 do not survive, the headline gain is due to future leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"At action time a POMDP agent may use only observations up to the current step. The paper's Appendix defines a backward GRU (Eqs. 12, 14, 16, 18) whose hidden state at t depends on h_{t+1}, and Algorithm 1 applies `self.biGRU(x)` to the whole input x in one pass. If x is the full episode or a context window whose later positions are future relative to an intermediate decision, then the Q-value at position t (Eq. 23) is contaminated by future observations. The text claims Q-values are predicted 'at each timestep' and the diagram shows Q1..Qt, so this is not a remote possibility. If this leakage exists, the large reported gains (87.39% over DTQN, 482.04% over DRQN) would be an artifact of lookahead, not of better memory. Conversely, if the authors always feed O1..Ot and read only the final token, the model is causal, but that procedure is never stated. Additionally, Table 1 does not reproduce the claimed margins under a standard mean-of-ratios, ratio-of-means, or percentage-point aggregation, so the quantitative central claim lacks a clear basis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes DBGFQN, a deep Q-network that replaces the feed-forward network in a transformer encoder with a single bidirectional GRU layer, and evaluates it on 23 POMDP-style environments against DRQN, DTQN, and several uni- and bi-directional recurrent variants. The authors claim that this architecture swap improves success rates by margins ranging from 87.39% to 482.04% on average across the 23 environments and reduces the parameter count by about 25% compared with traditional transformers. The paper also reports ablations addressing five research questions about the necessity of feed-forward expansion, the choice of recurrent layer, environment-dependent behavior, and a hallucinated-structure experiment.","tokens_in":12248,"tokens_out":6326,"duration_ms":55064,"significance":"The question of whether a recurrent layer can replace the feed-forward network in transformer-based POMDP agents is timely and practically relevant, and the paper contains useful infrastructure: a 23-environment benchmark, forward-pass pseudocode, and a hyperparameter table. If the causal-inference ambiguity were resolved and the reported margins were reproducible from the tables, the architecture would be a meaningful contribution. As presented, however, the central quantitative claims are not substantiated: the bidirectional GRU as described may leak future observations into the Q-values used for action selection, and the headline improvement figures are not reproducible from Table 1. The manuscript therefore needs substantial clarification and re-analysis before its claims can be assessed.","major_comments":[{"comment":"The bidirectional GRU is defined in the Appendix with a backward pass whose hidden state at time t depends on h_{t+1}, and Algorithm 1 applies self.biGRU(x) to the entire context window x in a single call. The text and Figure 1 show Q-values at every timestep (Eq. (23), Q_t), so if the action at time t is selected using Q_t computed from the full context window, the backward pass leaks information from observations O_{t+1..K}. The paper never states that actions are selected only from the final token of the BiGRU output or that a causal mask is applied. This is a load-bearing ambiguity: without a causal inference procedure, the reported gains may be an artifact of lookahead rather than improved memory. The authors must specify the exact inference-time protocol and empirically verify that no future information is used.","section":"§3, Appendix Eqs. (12), (14), (16), (18), Algorithm 1"},{"comment":"The headline claim of an 87.39% average improvement over DTQN is not reproducible from Table 1. Averaging the success rates over the 23 environments gives approximately 0.466 for DBGFQN versus 0.365 for DTQN, a relative improvement of roughly 28%, not 87.39%. The paper does not state whether the reported margin is a mean of per-environment ratios, a ratio of means, a percentage-point difference, or some other aggregation. The authors should report the aggregation formula, per-environment margins, standard errors, and either the code or raw results needed to verify the 87.39% and 482.04% figures.","section":"§5.3 and Table 1"},{"comment":"The claim that DBGFQN reduces the overall parameter count by 25% compared with traditional transformers is not supported by Table 3, which lists parameter counts for DTQN1-DTQN3, DRFQN, DLFQN, DGFQN, DBRFQN, and DBLFQN, but no DBGFQN column. Moreover, the DGFQN counts in Table 3 (e.g., 97,285 for Hallway) are comparable to DTQN3 (97,029), not 25% smaller. The authors should add the DBGFQN parameter count and state exactly which baseline and configuration yields the claimed 25% reduction.","section":"§1 and Table 3"},{"comment":"The paper claims improved sample efficiency, but Table 1 reports only a final mean success rate and the figures are not described in enough detail to show learning curves or fixed-budget comparisons. A single scalar success rate cannot demonstrate that DBGFQN reaches a given performance level with fewer interactions. The authors should present learning curves or interaction-budget comparisons, ideally with variance across seeds, to support the sample-efficiency claim.","section":"§4 and §5.3"}],"minor_comments":[{"comment":"The captions for Figures 5-7 contain a typo ('Comparitive') and the environment ranges are inconsistent: Figure 7 is titled 'environments 7-23' while the preceding figures cover 1-8 and 9-16; please clarify the intended ranges.","section":"Figure captions"},{"comment":"The softmax notation in Eq. (7) is unclear: the summation index and upper limit appear garbled (P over O_{t+1} o=1), and the notation should be rewritten as a standard softmax over the context positions.","section":"Appendix Eq. (7)"},{"comment":"There are grammatical errors, including 'the feed-forward layers was replaced' in §5.2 and 'have also gaining popularity' in §2; these should be corrected.","section":"§5.2 and §2"},{"comment":"Table 3 repeats identical parameter counts for different Gridverse sizes (e.g., GV 5x5, GV 7x7, GV 9x9), which is confusing; clarify whether the counts are environment-specific or the same architecture weights.","section":"Table 3"},{"comment":"The paper introduces RQ4 but then moves to RQ5 without an explicit subsection that answers RQ4; the discussion in §5.3 should be explicitly connected to RQ4.","section":"§5.3 and §5.4"},{"comment":"The hallucination experiment is described only qualitatively; please report the actual numbers behind Figure 8 and clarify whether the same artificial-room layout and training budget were used for all models compared.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The causal-leakage concern in the major comments is serious: the architecture as written and sketched could easily allow the backward GRU to condition on future observations. I would ask the editor to require an explicit description and empirical verification of the inference-time procedure before considering the paper further. The missing DBGFQN column in Table 3 and the non-reproducible headline margins are also blockers that should be fixed in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is simple and worth thinking about: replace the transformer FFN with a single BiGRU layer, and study when that helps in POMDPs. That is a sensible, low-cost architectural intervention, and the paper is honest enough to investigate it as a research question rather than just dropping a new module. The RQ-driven analysis, especially the hallucination experiment in Section 5.4, is a genuinely nice touch: it tries to causally test the claim that structural cues drive the benefit of recurrence. That is more than most architecture papers do.\n\nThe problem is that the headline numbers do not hold up. The abstract and Section 5.3 claim an average improvement of 87.39% over DTQN, but Table 1 gives mean success rates of about 0.466 for DBGFQN and 0.365 for DTQN, which is roughly 28% relative improvement under any standard aggregation. The 482.04% claim over DRQN also does not survive contact with Table 1. No aggregation formula is given, so the reader cannot tell what was averaged or how. That is a load-bearing flaw because the entire significance story rests on those margins.\n\nThe parameter-count claim is also shaky. Section 1 says the model reduces parameters by 25% compared to traditional transformers, but Table 3 shows that DGFQN has more parameters than DTQN3 in every listed environment (e.g., 366,662 vs 366,150 for Hallway). Since DBGFQN is presumably DGFQN plus an extra direction, it cannot be 25% smaller. The authors may have compared to a different transformer variant, but they do not say so.\n\nThere is also the causality question the stress-test note raises. The backward GRU is defined with h_{t+1}, and Algorithm 1 applies biGRU to the entire input. If Q-values are read at intermediate timesteps, then future observations leak into current decisions, and the reported gains could be lookahead rather than memory. If the authors only read the final token, they need to say so. The paper never does. This is not a minor omission; it is central to whether the empirical result means anything.\n\nThe work is not fundamentally unserious. The architecture is well-motivated, the writing is clear, and the environment taxonomy is a useful starting point. But the evaluation is not currently reproducible, and the causality issue is unresolved. This is the kind of paper that deserves a serious referee, but it needs heavy revision: report seed-level results, define the aggregation, correct the parameter count, and specify the exact inference-time procedure. If those are fixed, the empirical claim can be evaluated fairly. As it stands, I would not cite the headline numbers, and I would not pass it without those changes.","headline":"A plausible architectural tweak with an unreproducible headline number and an unaddressed causality question; worth a serious referee after the authors fix the evaluation.","tokens_in":12776,"tokens_out":975,"would_cite":false,"duration_ms":10547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that replacing a transformer's feed-forward network with a single bidirectional GRU layer produces a more sample-efficient and 25% smaller Q-network for partially observable RL, with success-rate improvements of 87–482%…","keywords":["partially observable Markov decision processes","reinforcement learning","transformer","bidirectional gated recurrent unit","sample efficiency","Q-network","gridverse","recurrent neural networks"],"falsifier":"Re-run the 23-environment benchmark with a strictly causal variant of DBGFQN — the same BiGRU block but with the backward pass masked so it never sees observations after the decision timestep — and check whether the 87.39% average gain over DTQN survives; if it collapses, the reported margins depend on future leakage rather than on genuine memory.","tokens_in":11788,"feed_emoji":"🔁","tokens_out":5575,"duration_ms":51720,"temperature":0.7,"pith_summary":"Partially observable environments force an RL agent to remember and recombine past observations, and transformers deliver that memory at a high parameter cost. This paper claims that the transformer's multi-layer feed-forward block can be replaced by a single layer of bidirectional gated recurrent units, producing the Deep BiGRUformer Q-Network (DBGFQN). Across 23 POMDP benchmarks, the network is reported to outperform transformer-, attention-, and recurrence-based baselines by 87.39% to 482.04% on average, while cutting total parameters by about 25%. If true, this would make memory-equipped RL agents substantially more parameter-efficient and improve their sample efficiency in partially observable settings.","feed_headline":"BiGRU swap lifts POMDP success by up to 482%","feed_subtitle":"One recurrent layer beats deep feed-forward blocks across 23 POMDP tasks at 25 percent fewer parameters.","key_machinery":"The load-bearing mechanism is the BiGRUformer encoder block: self-attention over a context of K observations, followed by layer normalization, then a single bidirectional GRU that reads the sequence forward and backward, followed by residual addition and a second layer normalization, with a final linear map to Q-values. The BiGRU's two hidden-state updates — reset gate, update gate, and candidate state in both directions — let the network combine forward context (past observations) with backward context (later observations in the context window) before Q-values are computed. It works because the backward pass acts as a learned attention-like summarizer over the whole window, while the forward pass preserves causal order; together they refine the self-attention representation before the value head.","core_discovery":"The central claim is that feed-forward expansion inside a transformer Q-network is not necessary for partial observability: a single BiGRU layer placed after multi-head self-attention captures the temporal dependencies that POMDP agents need, and does so better than the feed-forward blocks it replaces. The paper further claims that this design beats DTQN by 87.39%, DGFQN by 96.14%, and DRQN by 482.04% on average over 23 environments, and that bidirectional recurrence matters most in environments with dense structure and strong conditional dependencies, such as Gridverse Memory Four Rooms and Keydoor. The authors therefore conclude that POMDPs divide into classes for which different architectural choices are appropriate, and that large feed-forward expansions can be abandoned in favor of a compact recurrent layer.","pith_inferences":["If the Q-values for action selection are read only from the final timestep of the BiGRU-processed context, the backward pass is safely causal; this is the natural deployment rule, but the paper should state it explicitly because intermediate-timestep outputs would leak future observations.","The claimed margins of 87.39% and 482.04% depend on an unspecified aggregation rule across 23 environments; recomputing them as medians or per-environment geometric means would show whether the gains are consistent or concentrated in a few tasks.","The hallucination experiment suggests a practical diagnostic: injecting fake structure into a new POMDP could predict whether a recurrent agent will beat a feed-forward one before training the full model."],"forward_implications":["POMDP agents can get transformer-level memory at roughly 25% fewer parameters, making memory-equipped policies more practical for edge devices and real-time control.","The standard 4x embedding-dimension feed-forward expansion is not essential; a single recurrent layer can replace it, which may simplify transformer design in RL.","Performance differences across the 23 environments indicate that POMDPs should be classified by structural density and conditional dependencies when choosing between recurrent and non-recurrent architectures.","Artificially adding structure (hallucinated rooms) to a memory-light grid improves base-model performance, suggesting that environment structure itself drives the benefit of recurrence."],"supporting_citations":[{"why":"Defines DTQN, the transformer Q-network baseline and the environment suite the new model is compared against.","marker":"[23]"},{"why":"Introduces DRQN, the recurrent LSTM baseline that motivates replacing feed-forward layers with recurrence.","marker":"[24]"},{"why":"Shows bidirectional recurrent networks in deep RL, the precedent for the BiGRU module adopted here.","marker":"[27]"},{"why":"Supplies the original transformer architecture whose feed-forward block is replaced.","marker":"[30]"},{"why":"Argues vanilla transformers underperform in POMDPs, motivating the hybrid recurrent-transformer design.","marker":"[55]"},{"why":"Provides the DQN formulation that the Q-network training loop builds on.","marker":"[3]"}],"fun_headline_variants":["BiGRU layer beats feed-forward blocks in POMDPs","Single recurrent layer tops transformer in POMDP tasks","BiGRU swap lifts POMDP scores by up to 482%","Leaner transformer with BiGRU improves POMDP sample efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the agent's action is chosen from Q-values computed only at the final timestep of the BiGRU-processed context; if intermediate outputs are used, the backward GRU leaks future observations into the decision and the reported gains would not reflect a valid POMDP policy.","fun_headline_variants_meta":{"raw":{"variants":["BiGRU layer beats feed-forward blocks in POMDPs","Single recurrent layer tops transformer in POMDP tasks","BiGRU swap lifts POMDP scores by up to 482%","Leaner transformer with BiGRU improves POMDP sample efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000698,"raw_usage":{"total_tokens":3149,"prompt_tokens":939,"completion_tokens":2210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2136}},"tokens_in":555,"tokens_out":2210,"duration_ms":16976,"temperature":1.0,"reasoning_tokens":2136,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:56:51.925604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 23-environment benchmark with a strictly causal variant of DBGFQN — the same BiGRU block but with the backward pass masked so it never sees observations after the decision timestep — and check whether the 87.39% average gain over DTQN survives; if it collapses, the reported margins depend on future leakage rather than on genuine memory.","supporting_citations":[{"cited_title":"Deep reinforcement learning with bidirectional recurrent neural networks for dynamic spectrum access,","cited_arxiv_id":null,"evidence_quote":"Shows bidirectional recurrent networks in deep RL, the precedent for the BiGRU module adopted here."},{"cited_title":"Rethinking transformers in solving pomdps,","cited_arxiv_id":null,"evidence_quote":"Argues vanilla transformers underperform in POMDPs, motivating the hybrid recurrent-transformer design."}],"review_version":1}