{"id":"9fcae8a3-3593-4242-93c9-f76aff9ae430","arxiv_id":"2508.00046","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A JAX benchmark suite of nine memory-improvable partially observable RL environments, with evidence that recurrent and transformer agents beat memoryless baselines but fall short of state-augmented ceilings.","lead":"POBAX is a new open-source benchmark suite that tests whether reinforcement learning agents can use memory when the world is only partially visible, such as when a robot cannot see a goal or an inventory. The paper defines memory improvability as a performance gap between agents with less and more information, and shows that three memory-based algorithms improve on these tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Architecture mismatch between the observation-only floor and the full-state ceiling confounds the memory improvability gap in five of nine POBAX environments.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the full-state ceiling and observation-only floor differ in network architecture for five of nine environments, confounding the memory improvability measurement. This is the most serious threat to the paper's central empirical claim because memory improvability is defined by a gap 'all other factors roughly equal,' and an architecture mismatch is a direct violation of that condition. The concern is concrete, technical, and testable. I agree with the CONDITIONAL verdict: the benchmark suite, JAX implementation, environment coverage, and confounding-factor ablations are valuable contributions, but the headline claim that all POBAX environments are memory improvable needs architecture-matched baselines to be fully supported. The paper's own appendix disclosures strengthen the case for a conditional rather than a reject verdict, since the authors are transparent about the limitation; however, the main text's unqualified claim overstates what the current experiments demonstrate.","tokens_in":21480,"tokens_out":2829,"duration_ms":27832,"concrete_test":"Re-run the five disputed environments with four matched conditions using identical hyperparameters and training budgets: (i) memoryless MLP on partial observations, (ii) memoryless MLP on full-state/perfect-memory observations, (iii) RNN on partial observations, and (iv) RNN on full-state/perfect-memory observations. If (ii) exceeds (i) by the same margin as in Figure 6, and (iii) does not already reach (ii), the memory improvability gap is genuine and information-driven. If (ii) approximately equals (i), or (iii) approximately equals (ii), the current reported gap is an artifact of network architecture rather than state information.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that all POBAX environments are memory improvable—rests on the Section 4 definition requiring a performance gap between agents with more or less state information 'all other factors roughly equal.' For RockSample(11,11), RockSample(15,15), Walker-V, HalfCheetah-V, and No-inventory Crafter, the 'full state' or 'perfect memory' ceiling is trained with an RNN or transformer, while the observation-only floor is a memoryless MLP (Appendices C.3, C.5, C.8). This violates the 'roughly equal' condition: the observed gap could stem from greater function-approximation capacity rather than from additional state information. The paper discloses this asymmetry in the appendix but not in the main results, and the abstract's stronger claim that all tasks are memory improvable is therefore not established for these five configurations. The remaining four environments—T-Maze, Battleship, DMLab MiniGrid-01/02, and the two Visual Mujoco tasks—do not share this confound, but the suite-level conclusion depends on the disputed five.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that a partial-observability benchmark should satisfy two properties: coverage of diverse forms of state aliasing, and a memory-improvability signal defined as a performance gap between agents with more versus less state information when other factors are held roughly equal. It introduces POBAX, a JAX-based suite containing T-Maze, RockSample(11,11), RockSample(15,15), Battleship, Walker-V, HalfCheetah-V, two DeepMind Lab MiniGrid mazes, two visual Mujoco tasks, and No-inventory Crafter. The paper categorizes partial observability into eight forms, reports ablations on the number of parallel environments and network width, recommends per-environment hyperparameters, and evaluates recurrent PPO, λ-discrepancy PPO, and Transformer-XL with 30-seed confidence intervals. The central claim is that all POBAX environments are memory improvable and require hard-to-learn memory functions.","tokens_in":21696,"tokens_out":8227,"duration_ms":79456,"significance":"If the central claim holds, POBAX would be a valuable community resource: it covers several distinct forms of partial observability, is implemented entirely in JAX for GPU-scalable experimentation, ships with recommended hyperparameters, and provides extensive baselines. The paper's strengths include the open-source release, the detailed appendices, the systematic hyperparameter sweeps, the 30-seed confidence intervals, and the explicit investigation of confounding factors such as parallel-env count and network width. The timing experiments in Appendix A also support the scalability argument. However, the suite-level memory-improvability claim is only as strong as the floor/ceiling comparisons, and several of those comparisons do not satisfy the paper's own 'all other factors roughly equal' condition.","major_comments":[{"comment":"The central claim that all POBAX environments are memory improvable is not established for five of the configurations. For RockSample(11,11) and RockSample(15,15), Appendix C.3 states that the perfect-memory agent was trained with an RNN rather than a memoryless MLP; for Walker-V and HalfCheetah-V, Appendix C.5 states that an RNN was used for the fully observable results; and for No-inventory Crafter, Section 6.2 states that the 'full state' agent is a transformer trained on regular Crafter observations with the inventory included. The observation-only floors in Figure 6 are memoryless MLPs. Under the Section 4 definition, memory improvability requires a gap with 'all other factors roughly equal,' and an RNN or transformer ceiling does not satisfy that condition, since the gap may reflect greater function-approximation capacity rather than additional state information. The appendix disclosures do not repair the abstract's unqualified claim. Please rerun these ceilings with a memoryless MLP, or otherwise match the architecture between floor and ceiling, or restrict the memory-improvable claim to the configurations without this confound.","section":"§6.1, §6.2, App. C.3/C.5/C.8"},{"comment":"For Battleship, the green 'STATE' ceiling in Figure 6 is not a neural agent trained with more state information; it is the mean return of an optimal belief-state policy computed programmatically. The floor is a memoryless PPO agent trained from hit/miss observations. The gap between an optimal solver and a sample-based learner can be large even in the absence of partial observability, due to sample complexity, exploration difficulty, or optimization error. This violates the Section 4 requirement that other factors be roughly equal, in a different way from the architecture mismatch. Please add a trained ceiling using the same PPO learning setup with full-state or perfect-memory observations, and report the optimal belief policy as a separate reference line rather than as the 'STATE' ceiling.","section":"§6.2, Fig. 6, App. C.4"},{"comment":"The abstract and Section 6 assert that the POBAX tasks require hard-to-learn memory functions, but the paper never defines or operationalizes hardness. The only supporting evidence offered is that the three memory-based algorithms improve over the observation-only floor and underperform the ceiling. Given the architecture and ceiling mismatches described above, that evidence does not establish hardness. Please provide an operational criterion for 'hard-to-learn'—for example, required memory length, information-theoretic complexity, or the number of training steps needed to close a fixed fraction of the gap—and report it per environment.","section":"Abstract, §6, Fig. 6"}],"minor_comments":[{"comment":"The phrase 'performance of a agents' should be 'performance of agents'.","section":"Abstract"},{"comment":"The sentence 'In Figure 3 investigate the effect of network width' is missing a subject; it should read 'In Figure 3 we investigate'.","section":"§3.2"},{"comment":"The sentence 'avoid all the back ones' appears to be a typo for 'avoid all the bad rocks.'","section":"App. C.3"},{"comment":"The exact masked features for each masked-control environment are not specified in the appendix; the text directs readers to the code repository. Since the information content of the floor and ceiling observations is central to the paper's claims, please list the observation vectors explicitly.","section":"App. C.5"},{"comment":"The architecture used for the fully observable ceiling in the visual Mujoco environments is not stated. Please specify explicitly whether that ceiling uses a memoryless MLP, an RNN, or a transformer.","section":"App. C.7"},{"comment":"For No-inventory Crafter, the curve labeled 'STATE' in Figure 6 is actually the regular Crafter observation with the inventory visible, not the underlying environment state; please relabel this curve in the figure and caption to avoid confusion.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"One of the three benchmarked algorithms, λ-discrepancy (Allen et al., 2024), is co-authored by an author of this manuscript. This is not a correctness issue, but the relationship should be disclosed prominently in the main text rather than appearing only in the references. The paper is otherwise a good fit for the journal's benchmark-oriented scope, provided the floor/ceiling confounds are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Memory improvability is a genuinely good selection criterion, and POBAX is a solid, well-engineered benchmark. But the headline claim that every POBAX environment is memory improvable is not fully established: for five of the eleven tasks, the floor is a memoryless MLP and the ceiling is an RNN or transformer, so the gap confounds architectural capacity with hidden state information.\n\nWhat is new and good: The memory improvability property—requiring a performance gap between agents with more vs. less state information, other factors equal—is a clean reframing of oracle-gap evaluation. The paper backs it with a useful taxonomy of partial observability, carefully documents confounders like number of parallel environments and network width, and ships a fast JAX implementation with recommended hyperparameters, 30-seed confidence intervals, and open code. The environment set covers localization, visual control, games, and pixel domains, including new variants like no-inventory Crafter. That is a real contribution to benchmarking practice.\n\nThe soft spots: The architecture mismatch is real and disclosed in the appendix (C.3, C.5, C.8) but not in the main results. For RockSample(11,11), RockSample(15,15), Walker-V, HalfCheetah-V, and no-inventory Crafter, the 'more state information' agent is not just more informed—it uses a strictly more expressive function class. For no-inventory Crafter the ceiling is a transformer on the original observation with inventory, not even a full-state representation. That means the observed gap could overstate memory improvability. The remaining tasks (T-Maze, Battleship, DMLab Minigrid 01/02, Visual Ant/HalfCheetah) are not affected by this confound, so the suite still has value. But the abstract's claim that 'these tasks are all memory improvable' is too strong for five of eleven.\n\nAlso minor: the paper leaves out two of its own eight PO categories (unknown opposition, needle-in-a-haystack), which is fine, but the coverage claim should be softened.\n\nIn sum, the paper is honest—it discloses these limitations—and the benchmark is useful. I'd send it to review, but the authors should be asked to either add architecture-matched baselines (e.g., an MLP on full state and an RNN on partial obs) or to restrict the memory improvability claim to the configurations where the comparison is clean.\n\nRecommendation: worth a serious referee; conditional accept after the confound is addressed.","headline":"A useful JAX benchmark and a sensible selection criterion, but the 'all environments memory improvable' claim is confounded by architecture mismatches in five of eleven tasks.","tokens_in":22258,"tokens_out":3623,"would_cite":true,"duration_ms":33682,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a usable partial-observability benchmark needs broad coverage across forms of state aliasing plus a clear floor-to-ceiling performance gap, and introduces POBAX, a JAX suite whose tasks are all memory improvable and…","keywords":["reinforcement learning","partial observability","POMDP","benchmarking","memory improvability","JAX","recurrent policies","state aliasing"],"falsifier":"Retrain the full-state ceiling agents for RockSample(11,11), RockSample(15,15), velocity-only Walker and HalfCheetah, and no-inventory Crafter with the same memoryless MLP used for the observation-only floor; if the performance gap largely collapses or reverses, those environments' memory improvability is an artifact of function approximation power rather than a signal about partial observability.","tokens_in":21257,"feed_emoji":"🧠","tokens_out":15306,"duration_ms":126637,"temperature":0.7,"pith_summary":"Mitigating partial observability is necessary for general reinforcement learning agents, but most existing benchmarks test only narrow forms of state aliasing such as feature masking and Gaussian noise, where measured gains can come from hyperparameter choices, network size, or learning dynamics instead of memory. The paper argues that a useful partially observable benchmark needs two properties: coverage across the many distinct forms of partial observability, and memory improvability, meaning a clear performance gap between agents with more and less state information while everything else is held roughly equal. It introduces POBAX, a JAX-based suite of environments spanning localization and mapping, visual occlusion, object tracking, moment features, games, and more. The central claim is that every POBAX environment is memory improvable and requires hard-to-learn memory functions, giving the field a concrete signal for judging whether an algorithm actually mitigates partial observability.","feed_headline":"Memory beats observation on every POBAX benchmark task","feed_subtitle":"POBAX's floor-ceiling gaps tell researchers when added memory is the real source of an agent's gains","key_machinery":"The mechanism that carries the argument is the memory improvability gap: the difference in return between an agent fed partial observations and an agent fed a richer state representation (or an optimal belief policy), with the learning algorithm, network size, and other hyperparameters held fixed. The formal setting is the POMDP, whose observation function $\\Phi : S \\to \\Delta\\Omega$ maps hidden states to incomplete observations, and whose memory functions $\\mu(h_t)$ condense the history $h_t = (o_t, a_t, \\ldots, o_0, a_0)$ into a fixed-size memory state. The paper also contributes an eight-category taxonomy of partial observability—noisy state features, visual occlusion, object uncertainty and tracking, spatial uncertainty, moment features, unknown opposition, episode nonstationarity, and needle-in-a-haystack tasks—and selects at least one representative environment per category so the suite has coverage as well as signal. The gap is the instrument: a large floor-to-ceiling distance means added memory is what would close the distance, so an improvement by a memory-based algorithm is attributable to mitigating partial observability rather than to a better function approximator.","core_discovery":"The paper's central claim is that all POBAX environments are memory improvable and require hard-to-learn memory functions. Memory improvability is the existence of a performance gap between an observations-only agent (the floor) and an agent that receives more state information (the ceiling), all other factors roughly equal; the paper verifies this gap with 30-seed experiments across T-Maze, RockSample(11,11), RockSample(15,15), Battleship, velocity-only Walker and HalfCheetah, two DeepMind Lab MiniGrid mazes, pixel-based Ant and HalfCheetah, and no-inventory Crafter. In every environment three memory-based algorithms—recurrent PPO, $\\lambda$-discrepancy with recurrent PPO, and Transformer-XL—improve on the observation-only agent and land below the full-state agent, which the authors read as evidence that the gains come from mitigating partial observability rather than from function approximation or hyperparameters. The suite is implemented entirely in JAX, and the paper supplies recommended per-environment hyperparameters so the floor and ceiling can be reproduced out of the box.","pith_inferences":["A natural extension is to make the memory-improvability test a gate for admitting new tasks to the benchmark: compute the floor-ceiling gap under a fixed architecture budget and keep only environments where the gap is large and positive.","The paper suggests, though it does not test, a stronger claim: an algorithm that performs well across all POBAX categories simultaneously—pixels, sparse rewards, long horizons, object tracking—should transfer better to real deployments, which usually mix several forms of partial observability at once.","A stricter benchmark variant would require the floor and ceiling to share the same network family; testing whether the improvability gaps survive that constraint would separate the information gap from the function-approximation gap.","If the paper's reasoning is right, a memory-algorithm result published without its floor and ceiling curves cannot be attributed to memory; making all three curves a reporting norm would make benchmark comparisons decisive."],"forward_implications":["Researchers can benchmark new history-based algorithms out of the box: POBAX ships recommended hyperparameters for every environment, and the floor-to-ceiling gap is the declared measure of progress.","An algorithm that substantially closes the memory improvability gap has positive evidence that its gains come from mitigating partial observability, because the gap is constructed to hold architecture and hyperparameters fixed.","The gap size becomes a diagnostic: a large gap tells a practitioner that adding an RNN, transformer, or other memory mechanism is likely to pay off, while a small gap signals that memory is not the bottleneck.","Because every environment runs in JAX and scales on GPU, researchers can afford the large hyperparameter sweeps the paper shows are needed for reliable floor and ceiling curves.","The eight-category taxonomy gives benchmark designers a coverage checklist when assembling or extending their own partial-observability suites."],"supporting_citations":[{"why":"Supplies the POMDP formalism, the observation function, and the Tiger-problem example that define partial observability for the paper.","marker":"(Kaelbling et al., 1998)"},{"why":"Source of the T-Maze environment used as the benchmark's sanity check for memory length.","marker":"(Bakker, 2001)"},{"why":"Source of the RockSample(11,11) and RockSample(15,15) tasks with their distance-dependent stochastic rock sensor.","marker":"(Smith & Simmons, 2004)"},{"why":"Source of the Battleship environment, whose optimal belief policy provides the full-state ceiling.","marker":"(Silver & Veness, 2010)"},{"why":"Source of the masked continuous-control tasks; the benchmark reuses the velocity-only Walker and HalfCheetah variants.","marker":"(Han et al., 2020)"},{"why":"Provides the DeepMind Lab maze layouts on which the Navix-MiniGrid maze environments are based.","marker":"(Beattie et al., 2016)"},{"why":"Source of the Crafter benchmark that no-inventory Crafter modifies by masking the inventory strip.","marker":"(Hafner, 2021)"},{"why":"One of the three evaluated algorithms; the lambda-discrepancy extension of recurrent PPO is tested across all POBAX environments.","marker":"(Allen et al., 2024)"},{"why":"Establishes that single-frame Atari is partially observable, the motivating example for why simple masking benchmarks are insufficient.","marker":"(Hausknecht & Stone, 2015)"},{"why":"Prior partial-observability benchmark (POPGym) whose contrasting Battleship results the paper explains through uncontrolled hyperparameters.","marker":"(Morad et al., 2023)"}],"fun_headline_variants":["All POBAX tasks are memory improvable, with clear floor-ceiling gaps","Memory-improvable benchmark: POBAX shows gains on every task","Floor-ceiling gaps in all POBAX tasks signal memory improvability","Every POBAX benchmark task rewards memory over observation alone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gap between the observation-only floor and the full-state ceiling measures hidden information rather than architectural power, because in several environments the ceiling agent uses a recurrent or transformer network while the floor agent uses a memoryless multilayer perceptron.","fun_headline_variants_meta":{"raw":{"variants":["All POBAX tasks are memory improvable, with clear floor-ceiling gaps","Memory-improvable benchmark: POBAX shows gains on every task","Floor-ceiling gaps in all POBAX tasks signal memory improvability","Every POBAX benchmark task rewards memory over observation alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000499,"raw_usage":{"total_tokens":2505,"prompt_tokens":1068,"completion_tokens":1437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":684,"tokens_out":1437,"duration_ms":12245,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:28:27.305741+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the full-state ceiling agents for RockSample(11,11), RockSample(15,15), velocity-only Walker and HalfCheetah, and no-inventory Crafter with the same memoryless MLP used for the observation-only floor; if the performance gap largely collapses or reverses, those environments' memory improvability is an artifact of function approximation power rather than a signal about partial observability.","supporting_citations":[{"cited_title":"Transformer-XL is a memory-augmented algorithm that extends from the conventional architecture of transformers by incorporating segment-level recurrence","cited_arxiv_id":null,"evidence_quote":"One of the three evaluated algorithms; the lambda-discrepancy extension of recurrent PPO is tested across all POBAX environments."},{"cited_title":"Deep recurrent q-learning for partially observable MDPs","cited_arxiv_id":null,"evidence_quote":"Establishes that single-frame Atari is partially observable, the motivating example for why simple masking benchmarks are insufficient."}],"review_version":1}