{"id":"0bf5c3e8-29e0-4873-be33-60771e235569","arxiv_id":"2411.11088","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Offline RL methods adapted to factorised action spaces match or beat behaviour cloning and are more scalable than atomic action representations.","lead":"This paper adapts four standard offline reinforcement learning methods to work with action spaces made of several independent sub-actions, and tests them on new benchmark datasets. If the approach holds, offline agents could be trained more cheaply and reliably for problems like robotics and healthcare where action choices are naturally factorised.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark datasets are all generated by DecQN/REValueD (factorised) policies, so the empirical gains may reflect a matched inductive bias rather than a general benefit of factorised offline RL.","rationale":"The reader's weakest assumption (that the true Q-function decomposes as a mean of per-sub-action utilities) is real, but the authors explicitly discuss it in Sections 4.1-4.2 and even give a drug-interaction example where it fails. My concern is distinct and, in my view, more load-bearing: the benchmark's data collection protocol makes the problem easier for factorised methods in a way that is not acknowledged anywhere in the paper. All datasets are collected from DecQN/REValueD policies, which are factorised by construction. Thus the per-sub-action behaviour cloning and value decomposition used by the proposed methods are perfectly matched to the behaviour policy that generated the data. If the data came from a policy with correlated sub-actions, per-sub-action cloning would be misspecified and the decomposed Q-target would be biased, so the reported improvements over BC and the graceful scaling may not transfer. This is not an internal inconsistency, but it does limit the external validity of the central empirical claim. A concrete fix is to add experiments with non-factorised data collection; until then the claims should be scoped to datasets produced by factorised behaviour policies. The paper remains conditionally acceptable, but the condition should include this additional validation.","tokens_in":29054,"tokens_out":7744,"duration_ms":143782,"concrete_test":"Generate new datasets for the same Maze and DMC tasks by training an atomic behaviour policy (e.g., a DQN with a softmax over global actions, or a policy with explicit sub-action dependencies) to the same expert/medium levels, then rerun DecQN-BCQ/CQL/IQL/OneStep and factorised BC. If the factorised methods no longer consistently outperform BC, or fail to extract expert-level policies from medium-expert data, the central empirical claim is an artifact of factorised data collection rather than a general property of offline RL in factorisable action spaces.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The benchmark datasets are all collected from DecQN/REValueD policies (Section 6, Appendix C). These policies are factorised by construction: each sub-action is selected by its own utility argmax, so the behaviour policy is a product of independent sub-policies. Consequently the per-sub-action behaviour cloning used by DecQN-BCQ/IQL/OneStep and the per-sub-action value decomposition in DecQN-CQL are exactly matched to the data-generating process. The paper's central empirical claim, that factorised offline methods outperform BC and scale gracefully, is therefore potentially an artifact of the benchmark's data-generation procedure: the experiment cannot separate the benefit of factorised value decomposition from the benefit of a factorised behaviour policy. No experiment uses datasets collected by a non-factorised policy (e.g., an atomic DQN or a joint policy with correlated sub-actions), which is precisely the setting where the Section 4 decomposition assumption and the coverage argument are most fragile. This confound is not flagged in the limitation sections, unlike the Q-decomposition caveat.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies offline reinforcement learning in factorised discrete action spaces. It adapts four offline algorithms (BCQ, CQL, IQL, OneStep) to the DecQN value-decomposition framework, introduces a benchmark suite of datasets built from Maze and DeepMind Control tasks, and reports experiments showing that the factorised offline methods generally outperform behaviour cloning and plain DecQN, extract near-expert policies from medium-expert datasets, and scale more gracefully than atomic DQN-CQL in a two-task case study. The authors also provide a theoretical argument, supported by a simulation in Appendix A, that factorised representations can reduce overestimation bias for out-of-distribution actions.","tokens_in":29302,"tokens_out":5347,"duration_ms":48266,"significance":"If the empirical claims hold beyond the current benchmark, this is a useful contribution: it is the first systematic study of offline RL in factorisable action spaces, it provides a new benchmark with open code and datasets, and it reports careful engineering details such as compute time, GPU usage, and a 10-seed check. The paper is also commendably explicit about the limitations of value decomposition (Sections 4.1-4.2). However, the central empirical claim is not fully established because all benchmark datasets are collected by factorised DecQN/REValueD policies, so the experiments cannot separate the benefit of factorised value decomposition from the benefit of a factorised behaviour policy. The absence of atomic or non-factorised data-generation controls is the main weakness.","major_comments":[{"comment":"All benchmark datasets are collected by policies trained with DecQN/REValueD (Section 6, Appendix C), i.e. the behaviour policy is factorised by construction and each sub-action is selected independently. The per-sub-action behaviour cloning in DecQN-BCQ/IQL/OneStep and the per-sub-action CQL regularisation are therefore matched exactly to the data-generating process. The empirical gains reported in Figures 5-7 and Tables 7-9 may reflect this matched inductive bias rather than a general benefit of factorised offline RL. The paper does not include any dataset collected by an atomic or non-factorised policy (e.g. DQN or a policy with correlated sub-actions), which is precisely the regime where the coverage argument of Section 4 is most fragile. Please add such control datasets for at least a subset of tasks, or substantially qualify the central empirical claim.","section":"§6, Appendix C"},{"comment":"The hyperparameters τ, α, and λ (Table 2) are selected by evaluating the final policy on the same environment/task used for the reported results. While the paper forbids variation across datasets within a task, the selection still uses test-task performance. This can inflate the apparent advantage of the tuned offline methods over behaviour cloning, which does not have these method-specific hyperparameters, and it weakens the cross-method comparisons (e.g. the claimed CQL versus IQL/OneStep differences in Figures 6-7). Please report a sensitivity analysis or select hyperparameters on a validation split using off-policy evaluation, or at least discuss the magnitude of this effect.","section":"Table 2, Appendix D"},{"comment":"The theoretical bias-reduction argument assumes that both the approximate and the true Q-function decompose exactly as in Eq. (2). Section 4.2 acknowledges that this can fail under strong sub-action interactions, but the paper does not measure the decomposition error on the benchmark tasks, so it is unclear whether the reduction in target-difference bias (Eqs. 6-7) outweighs the approximation error in practice. I suggest adding a Monte Carlo estimate of the true Q-values (as in Appendix F) and reporting the mean squared error of the mean-of-utilities approximation on several tasks, to support the claim that 'for the same dataset we can potentially reduce overestimation bias moving from an atomic action representation to a factorised one' (Section 4).","section":"§4, Eqs. (4)-(7), Appendix A"}],"minor_comments":[{"comment":"In the second maximum of Eq. (7), the error term is written as \\epsilon_in for out-of-distribution sub-actions; this should be \\epsilon_out.","section":"Eq. (7)"},{"comment":"The OneStep target in Algorithm 4 mixes state and next-state indices: y = r + 1/N \\sum_i \\sum_{a_i} \\pi^i_{\\phi_i}(a_i|s) U^i_{\\hat\\theta_i}(s', a'_i) should sum over next-state sub-actions a'_i and evaluate \\pi_{\\phi_i}(a'_i|s').","section":"Algorithm 4"},{"comment":"Table 7 contains a stray '± ± ± ±' in the Maze (Actuators=12) row headers, and the table caption for Table 3 has a typo ('c' at the end of the heading).","section":"Table 7"},{"comment":"The paper states that datasets are made available, but the URL points only to the code repository; please provide a direct link to the datasets for reproducibility.","section":"Section 6"},{"comment":"The notation \\epsilon_in(s, a_in^i) and \\epsilon_out(s, a_out^i) in Eqs. (6)-(7) is introduced informally; a one-sentence definition would improve readability.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The data-generation confound is the main issue; if the authors add control datasets collected by atomic or non-factorised policies and address the hyperparameter-selection concern, the paper could become a solid contribution. I would not recommend rejection, as the case study comparing DQN-CQL and DecQN-CQL is well executed and the new benchmark is potentially useful to the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful, well-scoped empirical study rather than a breakthrough. It adapts four offline algorithms to factorised action spaces using DecQN-style value decomposition, contributes a new public benchmark, and reports careful experiments with code and data. The case study comparing DQN-CQL to DecQN-CQL is the most convincing part: as the number of bins grows, the atomic version degrades in both score and compute, while the factorised version stays roughly flat. That is a concrete, reproducible result.\n\nThe main soft spot is the one the stress-test flags: every dataset in the main benchmark was collected from DecQN or REValueD policies, which are factorised by construction. So the empirical superiority of the factorised offline methods over behaviour cloning could in part just reflect a matched inductive bias between the behaviour policy and the learner. The paper does not include datasets collected by an atomic or otherwise non-factorised policy, so it cannot separate these effects. This is not fatal to the main claim—the paper is explicitly about factorised action spaces, and its case study does compare against an atomic baseline—but it is a real limitation that should be stated more prominently than it currently is.\n\nThe other soft spots are milder. Hyperparameters are tuned per task, though not per dataset quality within a task; that is typical for this kind of study but still inflates results. The theoretical section is heuristic: it assumes the true Q-function decomposes as a mean of per-sub-action utilities, and the paper itself concedes in Sections 4.1-4.2 that this holds only under weak inter-action dependence. The drug-interaction example is honest about the failure mode. Appendix A's simulation is a nice sanity check, not a proof.\n\nOn balance, the paper does what it claims: it provides a formative investigation, a reusable benchmark, and four working adaptations. The writing is clear, the limitations are acknowledged, and the code and data are public. I would send it to peer review, with a request for the authors to add at least one benchmark where data is generated by a non-factorised policy and to make the dataset-generation confound explicit in the main text.","headline":"A solid, honest first benchmark for offline RL with factorised actions, with the caveat that all data comes from factorised policies.","tokens_in":29800,"tokens_out":2890,"would_cite":true,"duration_ms":25996,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Offline reinforcement learning in factorisable action spaces can be made practical and scalable by decomposing global actions into per-sub-action utilities, which reduces overestimation bias and lets adapted offline methods match or beat…","keywords":["offline reinforcement learning","factorised action spaces","value decomposition","DecQN","overestimation bias","conservative Q-learning","offline benchmarks","discrete control"],"falsifier":"Train both DecQN-CQL and atomic DQN-CQL on a task where the reward depends on a strong nonlinear interaction between two sub-actions (for example, a reward that is positive only when both sub-actions match) using a medium-expert dataset; if the factorised method cannot recover expert behaviour while the atomic method does, the claimed bias-reduction advantage fails in exactly the regime the paper concedes is its limit.","tokens_in":28847,"feed_emoji":"🧩","tokens_out":6464,"duration_ms":165757,"temperature":0.7,"pith_summary":"This paper aims to show that offline reinforcement learning can work well in factorisable discrete action spaces—where a global action is a combination of independent sub-actions—by representing values with the DecQN decomposition rather than treating every combination as a separate atomic action. The authors argue that factorising turns many out-of-distribution global actions into in-distribution sub-actions, which should lower the overestimation bias that plagues offline Q-learning. To test this, they adapt four established offline methods (BCQ, CQL, IQL, and one-step RL) to the factorised setting and evaluate them on a new benchmark of maze and continuous-control tasks with datasets of varying quality. Their results show these factorised offline methods generally outperform behaviour cloning and plain DecQN, extract expert-level policies from medium-expert data in many tasks, and scale more gracefully than atomic DQN-CQL as the number of bins grows.","feed_headline":"Factorised actions cut offline RL bias and scale past atomic Q-learning","feed_subtitle":"DecQN-based offline methods beat behaviour cloning on a new benchmark and hold up as action spaces grow.","key_machinery":"The central object is the DecQN value-decomposition: each sub-action space $i$ has its own utility function $U^i_{\\theta_i}(s,a^i)$, and the global action value is the mean of these utilities, $Q_\\theta(s,a)=\\frac{1}{N}\\sum_i U^i_{\\theta_i}(s,a^i)$. This identity carries the argument by shrinking the number of values to learn from $\\prod_i n_i$ to $\\sum_i n_i$ and by making sub-action coverage the relevant statistic for overestimation bias, since a global action can be out-of-distribution while its constituent sub-actions are in-distribution. The offline adaptations (DecQN-BCQ, DecQN-CQL, DecQN-IQL, DecQN-OneStep) all apply their regularisation at the sub-action level, preserving this decomposition while constraining or penalising value estimates.","core_discovery":"The paper's central claim is that the DecQN value-decomposition, $Q_\\theta(s,a) = \\frac{1}{N}\\sum_{i=1}^N U^i_{\\theta_i}(s,a^i)$, is a sound foundation for offline RL in factorisable action spaces because it converts the value-learning problem from a product of sub-actions into a sum, and because a global action that is absent from the dataset can still be composed of sub-actions that are present. That coverage advantage reduces the expected overestimation bias in target values compared with an atomic representation, especially when datasets are small or action spaces large. The authors support this with a noise model in which in-distribution errors are smaller than out-of-distribution errors, and they demonstrate empirically that DecQN-CQL retains performance and low Q-value error as the number of bins increases while DQN-CQL degrades sharply, and that factorised offline methods generally beat behavioural cloning and plain DecQN across their new benchmark.","pith_inferences":["A direct extension the paper leaves implicit is that factorisation should help most when datasets are small and sub-action coverage is high; this can be tested by sweeping dataset size and bin count while holding the task fixed.","The same bias-reduction logic suggests factorised offline methods could transfer to high-dimensional discrete control or healthcare dosing, but only in regimes where inter-action dependence is weak; strongly interacting settings would need a decomposition that models interactions rather than averaging.","The random-medium-expert results indicate a performance ceiling on complex tasks; combining factorisation with uncertainty estimation or ensembles, as has been done in continuous offline RL, is a natural next step that the paper does not itself test."],"forward_implications":["Using the same dataset, switching from an atomic to a factorised action representation can reduce expected overestimation bias, which is the main failure mode in offline RL.","Factorised offline methods (DecQN-CQL, DecQN-IQL, DecQN-OneStep) consistently match or beat behaviour cloning, and in many tasks they extract expert-level policies from medium-expert datasets.","As the number of sub-actions per dimension grows, DecQN-CQL stays near expert performance while atomic DQN-CQL degrades and its training time and GPU memory explode.","On low-quality or mixed datasets for complex control tasks, all factorised methods still improve over behaviour cloning, but their gap to expert performance remains visible, pointing to a limit of the approach."],"supporting_citations":[{"why":"It supplies the DecQN value-decomposition formulation, $Q_\\theta(s,a)=\\frac{1}{N}\\sum_i U^i(s,a^i)$, that the paper adapts to the offline setting.","marker":"[Seyde et al., 2022]"},{"why":"It provides the REValueD extension and the bias/variance comparison of DecQN versus DQN target differences that Section 4 builds on.","marker":"[Ireland and Montana, 2023]"},{"why":"It supplies the function-approximation noise model and the result that overestimation grows with the number of actions, which motivates the factorisation argument.","marker":"[Thrun and Schwartz, 1993]"},{"why":"It contributes the BCQ policy-constraint technique and the offline overestimation-bias framing that the paper adapts as DecQN-BCQ.","marker":"[Fujimoto et al., 2019b]"},{"why":"It contributes the conservative Q-learning loss that the paper adapts at the sub-action level as DecQN-CQL.","marker":"[Kumar et al., 2020]"},{"why":"It contributes implicit Q-learning via expectile regression, which the paper adapts as DecQN-IQL.","marker":"[Kostrikov et al., 2021b]"},{"why":"It contributes the one-step RL idea that the paper extends to the factorised setting as DecQN-OneStep.","marker":"[Brandfonbrener et al., 2021]"},{"why":"It provides the Maze environment used for the benchmark's factorisable action space.","marker":"[Chandak et al., 2019]"},{"why":"It supplies the D4RL-style dataset construction procedure (expert, medium, medium-expert, random-medium-expert) that the paper follows.","marker":"[Fu et al., 2020]"}],"fun_headline_variants":["Factorise actions to cut offline RL bias and scale past atomic Q-learning","Offline RL with factorised actions beats clones and scales better","DecQN-based offline RL: factorised actions reduce bias, scale well","New benchmark shows factorised offline RL outperforms behaviour cloning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the true Q-value of a global action is well approximated by the average of per-sub-action utilities; if sub-actions interact strongly, that average can be systematically wrong, and the factorised offline methods inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["Factorise actions to cut offline RL bias and scale past atomic Q-learning","Offline RL with factorised actions beats clones and scales better","DecQN-based offline RL: factorised actions reduce bias, scale well","New benchmark shows factorised offline RL outperforms behaviour cloning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000604,"raw_usage":{"total_tokens":2806,"prompt_tokens":923,"completion_tokens":1883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1808}},"tokens_in":539,"tokens_out":1883,"duration_ms":17025,"temperature":1.0,"reasoning_tokens":1808,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:55:16.063292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train both DecQN-CQL and atomic DQN-CQL on a task where the reward depends on a strong nonlinear interaction between two sub-actions (for example, a reward that is positive only when both sub-actions match) using a medium-expert dataset; if the factorised method cannot recover expert behaviour while the atomic method does, the claimed bias-reduction advantage fails in exactly the regime the paper concedes is its limit.","supporting_citations":[{"cited_title":"Solving continuous control via Q-learning","cited_arxiv_id":null,"evidence_quote":"It supplies the DecQN value-decomposition formulation, $Q_\\theta(s,a)=\\frac{1}{N}\\sum_i U^i(s,a^i)$, that the paper adapts to the offline setting."},{"cited_title":"Revalued: Regularised ensemble value-decomposition for factorisable Markov decision processes","cited_arxiv_id":null,"evidence_quote":"It provides the REValueD extension and the bias/variance comparison of DecQN versus DQN target differences that Section 4 builds on."},{"cited_title":"Issues in using function approximation for reinforcement learning","cited_arxiv_id":null,"evidence_quote":"It supplies the function-approximation noise model and the result that overestimation grows with the number of actions, which motivates the factorisation argument."},{"cited_title":"Conservative Q-learning for offline reinforcement learning","cited_arxiv_id":null,"evidence_quote":"It contributes the conservative Q-learning loss that the paper adapts at the sub-action level as DecQN-CQL."},{"cited_title":"Offline RL without off-policy evaluation","cited_arxiv_id":null,"evidence_quote":"It contributes the one-step RL idea that the paper extends to the factorised setting as DecQN-OneStep."},{"cited_title":"Learning action representations for reinforcement learning","cited_arxiv_id":null,"evidence_quote":"It provides the Maze environment used for the benchmark's factorisable action space."}],"review_version":1}