{"id":"9c6f7929-a719-4c03-b585-5694481c222a","arxiv_id":"2412.18946","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CAPS trains multiple policies with shared features and switches among them at test time to satisfy arbitrary cost constraints, beating prior offline safe RL baselines on 38 benchmark tasks.","lead":"This paper introduces CAPS, a method that trains several safe-driving policies at once and then picks the best one at each moment to meet a changing safety budget. It is useful for offline reinforcement learning systems that must obey different cost limits after deployment without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '34/38 safe' headline is computed by averaging normalized cost over three different cost thresholds; per-threshold safety is much lower (e.g., 18/38 at the {5,10} budget), so constraint satisfaction is not established as claimed.","rationale":"The reader's weakest assumption points to cost Q-function accuracy, which is a real theoretical gap: Theorem 1 assumes a perfect Qc and a cost-optimal πc, and the paper does not bound estimation error. My stress test does not dispute that concern. However, the more directly load-bearing weakness is in the empirical claim that the paper actually reports. Table 2 already provides strong evidence that safety counts depend heavily on the cost budget: CAPS(IQL) is safe in 18/38 tasks at {5,10}, 28/38 at {10,20}, and 33/38 at {20,40}. The main table's 'safe agent' criterion is an average normalized cost across three thresholds, which can label a task safe even when one threshold is violated. Since the abstract and Section 6.2 use '34/38 tasks' as the headline safety result, this aggregation issue affects the central claim directly. The fix is simple and concrete: report per-threshold normalized costs and count safety only when all thresholds are met. This is not a rejection of the framework; CAPS is a plausible wrapper and the shared-representation and FQE ablations are informative. But the empirical safety advantage, as currently quantified, is not yet established at the level claimed. A conditional acceptance is appropriate, with the per-threshold breakdown as a required revision rather than a stylistic improvement.","tokens_in":32610,"tokens_out":6145,"duration_ms":62254,"concrete_test":"Using the released code and DSRL datasets, recompute for every task and seed the normalized cost separately at each of the three thresholds used in Table 1 (and at the stricter budgets from Table 2, such as {5,10} and {10,20}). Count a task as safe only if its normalized cost is ≤ 1 at every evaluated threshold. If the per-threshold safe counts are 18/38, 28/38, and 33/38 while Table 1 reports 34/38, the aggregate count is an artifact of averaging and the 'satisfies the constraint in 34/38 tasks' claim must be revised to a per-threshold statement with the actual numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that CAPS(IQL) satisfies the cost constraint in 34/38 tasks and achieves the highest reward among safe methods in 18/38 tasks (Section 6.2, Table 1). The Table 1 caption states that each value is averaged over 3 distinct cost thresholds and marks a method as safe whenever its averaged normalized cost is ≤ 1. That safe labeling does not imply that the cost constraint is satisfied at every evaluated threshold. Table 2 shows exactly why this matters: CAPS(IQL) is safe in only 18/38 tasks at the {5,10} budget, 28/38 at {10,20}, and 33/38 at {20,40}. Because the paper's own per-threshold counts drop sharply as budgets tighten, the 34/38 number in Table 1 can be dominated by the easier high-threshold configurations and may conceal violations at stricter thresholds. The abstract's 'consistently outperforms' and the reader's strongest claim inherit this ambiguity. The theoretical section does not repair this gap: Theorem 1 assumes a perfectly estimated cost Q-function Qc and a cost-optimal fallback policy πc (Section 5), and the paper does not report any per-threshold verification. Thus the load-bearing condition for the empirical safety claim is not a particular value-function error model but a missing and feasible level of disaggregated reporting: normalized costs at each threshold separately, with a task counted as safe only if it satisfies every threshold.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAPS, a wrapper for offline safe reinforcement learning that trains K policies with a shared representation (reward-only, cost-only, and scalarized trade-offs) and switches among them at test time based on a learned cost Q-function filter and a learned reward Q-function selector. A finite-horizon safety bound is stated under assumptions of a perfect cost Q-function, an optimal-cost fallback policy, and limited optimal-cost variation. Experiments on 38 DSRL tasks compare CAPS(IQL) and CAPS(SAC+BC) with BC, BEAR-Lag, CPQ, COptiDICE, and CDT, with ablations on the number of policies, shared representation, and fitted Q-evaluation. The headline claims are that CAPS(IQL) is safe in 34/38 tasks and achieves the highest reward among safe methods in 18/38 tasks.","tokens_in":32965,"tokens_out":8821,"duration_ms":79498,"significance":"CAPS is a simple and potentially useful contribution: it reduces varying-constraint offline safe RL to standard offline RL, comes with public code, and reports extensive comparisons on 38 tasks with two base algorithms. The ablations on shared representations and number of heads are informative, and the conditional safety analysis is a reasonable starting point. However, the current manuscript does not fully substantiate the headline empirical claims because the main safety metric averages over cost thresholds, and the per-threshold counts in Table 2 are much lower at strict budgets. The theoretical guarantee also depends on an idealization (perfect Qc and cost-optimal πc) without an error-propagation version. These issues are fixable, and if addressed the paper would provide a strong wrapper-based baseline.","major_comments":[{"comment":"The headline claim that CAPS(IQL) is safe in 34/38 tasks is computed by averaging normalized cost over three distinct cost thresholds and labeling a method safe when this average is ≤ 1. This does not establish that the constraint is satisfied at any individual threshold. Table 2 shows that CAPS(IQL) is safe in only 18/38 tasks at the {5,10} budget, 28/38 at {10,20}, and 33/38 at {20,40}. Thus the 34/38 number conflates thresholds and is contradicted by the paper's own per-threshold counts at strict budgets. Please report normalized costs per threshold and define a task as safe only if it satisfies every evaluated threshold, or revise the abstract and Section 6.2 claims accordingly.","section":"Section 6.2, Tables 1 and 2"},{"comment":"There is a time-index inconsistency in the proof of Theorem 1. Assumption 2 bounds the variation of V^c_t(s') over next states s' after taking action a at state s and time t, but the next-state values entering the Bellman expansion should be V^c_{t+1}(s'), and the proof of Eq. (9) uses V^c_{t+1}(s') + c(s). As written, the per-support bound 'for any s'' does not follow from the stated assumption. The assumption should be re-indexed to V^c_{t+1}(s') (or the proof otherwise corrected), and the induction step should be checked under that corrected statement.","section":"Section 5, Assumption 2 and Appendix A, Eq. (9)"},{"comment":"The practical safety guarantee is conditional on a perfectly estimated cost Q-function Qc and a cost-optimal fallback policy πc. The paper says that extending the result to bounded estimation error is straightforward, but no such bound is provided. Since the CAPS filter in Eq. (3) can admit unsafe actions when Qc is inaccurate, the theorem as stated is an ideal-case analysis rather than a guarantee for the implemented system. Please provide a bounded-error version, or explicitly state this limitation and complement it with empirical evidence about the accuracy of the learned Qc.","section":"Section 5, Theorem 1 and CAPS filter (Eq. 3)"},{"comment":"The CDT comparison is confounded by hand-coded return targets. The manuscript itself notes that CDT requires return and cost targets and that the authors do not provide a clear methodology for selecting these targets; for additional cost configurations the authors interpolated or extrapolated the provided targets. Because Table 2's claim that CAPS 'consistently outperforms CDT across all cost limit configurations' relies on these extra configurations, the comparison is not fully controlled. Please provide a principled target-selection method for CDT or a sensitivity analysis over the target values used.","section":"Appendix B.3 and Table 2"}],"minor_comments":[{"comment":"The text says the algorithms are evaluated with 'three different target cost threshold configurations' but then lists only two sets, {10, 20, 40} and {20, 40, 80}; please clarify the intended configurations and their mapping to environment categories.","section":"Section 6.1"},{"comment":"The role of λk for the extreme policies πr and πc is described only in prose; the equation does not show that πr and πc correspond to the limiting cases λ=0 and λ→∞. Please state this explicitly and clarify how the λk values in Table 14 are selected.","section":"Section 4.2, Eqs. (5) and (9)"},{"comment":"The results are averaged over three seeds and twenty episodes, but no standard deviations or seed-level breakdowns are reported for the normalized rewards and costs. Reporting variance would help assess whether the safety counts and reward differences are stable.","section":"Tables 1 and 3-5"},{"comment":"Environment names are inconsistent across tables (for example, 'hardsparse' in Table 10 versus 'HardSparse' elsewhere); please standardize the naming.","section":"Appendix B.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the venue's scope, and the public code and broad benchmark evaluation are strengths. The main risk is not novelty but whether the empirical comparison is reported honestly: the threshold-averaging convention inflates the headline safety count, and the CDT comparison depends on hand-coded targets. Both issues are fixable within the manuscript. If the per-threshold results do not support the current claims, the conclusions should be weakened accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the CAPS paper. The core idea is genuinely useful: train a small set of policies (reward-only, cost-only, and scalarized mixtures) with a shared body, then at test time filter each policy's action through a learned cost Q-function and pick the highest-reward action among the safe ones. It reduces to two runs of an offline RL algorithm plus cheap policy extraction steps, and wrapping it around IQL and SAC+BC is straightforward. That is a real contribution, and the ablations—shared vs separate backbones, and policy count—are done carefully.\n\nThe experiments are broad: 38 DSRL tasks, multiple baselines, and per-threshold tables in the appendix. The method does appear to be safe more often than CDT, which is the only other method that handles varying cost limits. The theory is modest and honest: Theorem 1 gives a bound under a perfect cost Q-function and a cost-optimal fallback policy, with an extra (T-t)epsilon term. They explicitly note these assumptions, so that is not a hidden flaw.\n\nThe soft spots are mostly about presentation and claim strength, not the method itself. The abstract says CAPS 'consistently outperforms' existing methods, and the main table's lead claim is that CAPS(IQL) is safe on 34/38 tasks. But Table 1 marks a task safe when the normalized cost averaged over three cost thresholds is <= 1. The per-threshold counts in Table 2 are much lower at tight budgets: 18/38 at {5,10}. So the 34/38 headline is an average effect and can hide violations at stricter constraints. That is a real reporting problem, because safety is the primary metric. The fix is easy: report per-threshold safety counts, and treat a task as safe only if all thresholds are satisfied. Also, there are no error bars anywhere (they say three seeds, but no variance). The CDT comparison is complicated by hand-coded return targets, which they acknowledge in Appendix B.3; interpolating those targets for new cost sets could bias the baseline. Minor issue: the 'highest reward among safe methods in 18/38 tasks' is fine but should be read with the same per-threshold caveat.\n\nWho is this for? Anyone working on offline safe RL or constraint-conditioned deployment. The code is promised, and the wrapper is simple enough to adopt. It deserves a serious referee, but the revision should fix the safety accounting before publication.","headline":"Useful wrapper for offline safe RL with variable cost limits, but the headline safety count averages over thresholds and flatters the method.","tokens_in":33439,"tokens_out":2116,"would_cite":true,"duration_ms":19402,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that switching between a reward-maximizing and a cost-minimizing policy, filtered by learned cost Q-values, lets an offline-trained agent satisfy changing safety constraints at deployment without retraining.","keywords":["offline safe reinforcement learning","constraint-adaptive policy switching","varying cost constraints","cost Q-function filtering","policy switching","shared policy representation","implicit Q-learning","safety guarantees"],"falsifier":"Run CAPS on the 38 benchmark tasks with cost thresholds below every threshold used in training, and measure the fraction of episodes whose cumulative cost exceeds the cap; a sharp rise in violations would show the filter admits unsafe actions. More directly, build an MDP with a known large optimal-cost variation $\\epsilon$ and check whether realized total cost violates the bound $\\max\\{V^c(s), \\kappa-c_{<t}\\} + (T-t)\\epsilon$ for a $\\kappa$-admissible policy.","tokens_in":32434,"feed_emoji":"🛡️","tokens_out":9731,"duration_ms":74869,"temperature":0.7,"pith_summary":"Offline safe reinforcement learning usually trains one policy for one fixed cost threshold, so changing the safety budget at deployment means retraining. This paper proposes CAPS, a wrapper around any offline RL algorithm that trains a small set of policies with a shared representation—one reward-maximizing, one cost-minimizing, optionally more—and then, at each test-time state, filters each policy's proposed action through a learned cost Q-function $Q_c$ and picks the highest-reward action that still fits the current cost limit. The central claim is that this simple switching scheme adapts to varying cost constraints without retraining and is safer than existing offline safe RL methods. Across 38 benchmark tasks, CAPS wrapped around IQL satisfies the cost constraint in 34 of 38 tasks and reaches the highest reward among safe methods in 18 of 38 tasks. A theorem bounds expected total cost by $\\max\\{V^c(s), \\kappa-c_{<t}\\} + (T-t)\\epsilon$, where $\\epsilon$ measures the optimal-cost variation of the environment.","feed_headline":"Policy switching hits safety in 34 of 38 offline RL tasks","feed_subtitle":"Instead of retraining for each cost cap, a wrapper filters each action with a learned cost estimate and picks the safest high-reward option.","key_machinery":"The central mechanism is a filter-and-select loop driven by two learned Q-functions. $Q_c$ estimates expected future cost and acts as the safety filter; $Q_r$ estimates expected future reward and acts as the selector; the cost-minimizing policy $\\pi_c$ serves as a fallback when no proposed action is safe. Training uses a reduction to standard offline RL: train $Q_r$ and $Q_c$ from the dataset, extract $\\pi_r$ from $Q_r$, $\\pi_c$ from $Q_c$, and optionally intermediate policies from mixtures $Q_r - \\lambda_k Q_c$. A shared neural-network body with one output head per policy transfers features between the reward and cost objectives, and the paper shows this sharing is what makes the cost-minimizing head achieve meaningful reward and the reward head incur lower cost.","core_discovery":"This paper claims that a finite set of policies with a shared representation, switched by two Q-functions, is enough to handle unknown and varying cost constraints offline. At test time CAPS computes the actions proposed by all policies, keeps only those for which $Q_c(s,a) + c_{\\le t} \\le \\kappa$, and selects the kept action with the largest $Q_r(s,a)$; if none is kept it falls back to the cost-minimizing policy $\\pi_c$. The safety theorem states that any such policy is $\\kappa$-admissible and its expected remaining cost is at most $\\max\\{V^c_t(s), \\kappa-c_{<t}\\} + (T-t)\\epsilon$, assuming $Q_c$ is perfect and $\\pi_c$ is cost-optimal. The experimental claim is that this recipe is safe on 34/38 tasks with IQL and 30/38 with SAC+BC, and that shared representations are crucial to this performance.","pith_inferences":["Because the safety filter depends on the quality of $Q_c$, a natural extension is a calibration step that measures empirical violation rates across proposed cost limits before deployment and adjusts $\\kappa$ or the fallback policy accordingly.","The same filter-and-select recipe should generalize to multiple simultaneous constraints by replacing the scalar $Q_c$ with several cost Q-functions and requiring each constraint to pass the filter.","A learned fallback policy, rather than the pure cost minimizer, could raise reward under tighter budgets while preserving the guarantee, since the theorem only requires the fallback to be $\\kappa$-admissible.","The theorem's $(T-t)\\epsilon$ term identifies high-stochastic-cost environments as the stress test: deterministic or low-variation domains should show near-tight guarantees, while high-variation domains are where violation monitoring matters most."],"forward_implications":["One model can serve many deployment cost limits: changing $\\kappa$ at runtime only changes the filter, not the weights.","CAPS is algorithm-agnostic; any offline RL method that returns Q-functions and a policy can be wrapped, so base-algorithm improvements transfer directly.","The two-policy CAPS is a useful minimalist default: it removes scalarization hyperparameters and needs only two full offline RL runs.","If the environment's optimal-cost variation $\\epsilon$ is known, a user can choose $\\kappa$ in advance to satisfy a desired expected-cost bound.","Training cost stays low relative to per-threshold retraining: CAPS(IQL) trains in 24–33 minutes on the HalfCheetah task versus roughly 154 minutes for CDT, with more heads adding modest overhead."],"supporting_citations":[{"why":"IQL is the base offline RL algorithm for the CAPS(IQL) instantiation; it supplies both Q-functions and the policy extraction step.","marker":"(Kostrikov, Nair, and Levine 2022)"},{"why":"The TD3+BC-style framework motivates the SAC+BC instantiation with a behavior cloning regularization term for policy extraction.","marker":"(Fujimoto and Gu 2021)"},{"why":"Soft actor-critic is the off-policy RL component underlying the CAPS(SAC+BC) instantiation.","marker":"(Haarnoja et al. 2018)"},{"why":"Supplies the benchmark with 38 tasks and the normalized reward and cost evaluation protocol used for all comparisons.","marker":"(Liu et al. 2024)"},{"why":"CDT is the baseline that also handles varying cost constraints, and CAPS is compared against it across multiple cost limit configurations.","marker":"(Liu et al. 2023)"},{"why":"Fitted Q-evaluation is used in the ablation to compare off-policy-evaluation Q-functions against CAPS's offline-RL Q-functions.","marker":"(Le, Voloshin, and Yue 2019)"},{"why":"Cited for the observation that off-policy evaluation gives unreliable value estimates, which motivates using Q-functions produced by policy optimization.","marker":"(Figueiredo Prudencio, Maximo, and Colombini 2024)"}],"fun_headline_variants":["Offline safe RL adapts to new cost caps without retraining","Switching between learned policies meets safety in 34 of 38 offline tasks","Wrapper-based CAPS beats baselines across DSRL tasks","Two Q-functions enable constraint-adaptive policy switching in offline RL","CAPS: a wrapper for offline RL that obeys shifting cost limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learned cost Q-function $Q_c$ accurately estimates future costs and that the cost-minimizing policy $\\pi_c$ is truly near cost-optimal; the safety theorem assumes perfect estimation, and the empirical safety margin shrinks if that estimate is wrong.","fun_headline_variants_meta":{"raw":{"variants":["Offline safe RL adapts to new cost caps without retraining","Switching between learned policies meets safety in 34 of 38 offline tasks","Wrapper-based CAPS beats baselines across DSRL tasks","Two Q-functions enable constraint-adaptive policy switching in offline RL","CAPS: a wrapper for offline RL that obeys shifting cost limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3618,"prompt_tokens":896,"completion_tokens":2722,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":2630}},"tokens_in":512,"tokens_out":2722,"duration_ms":16457,"temperature":1.0,"reasoning_tokens":2630,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:18:27.555766+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CAPS on the 38 benchmark tasks with cost thresholds below every threshold used in training, and measure the fraction of episodes whose cumulative cost exceeds the cap; a sharp rise in violations would show the filter admits unsafe actions. More directly, build an MDP with a known large optimal-cost variation $\\epsilon$ and check whether realized total cost violates the bound $\\max\\{V^c(s), \\kappa-c_{<t}\\} + (T-t)\\epsilon$ for a $\\kappa$-admissible policy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IQL is the base offline RL algorithm for the CAPS(IQL) instantiation; it supplies both Q-functions and the policy extraction step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The TD3+BC-style framework motivates the SAC+BC instantiation with a behavior cloning regularization term for policy extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark with 38 tasks and the normalized reward and cost evaluation protocol used for all comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CDT is the baseline that also handles varying cost constraints, and CAPS is compared against it across multiple cost limit configurations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited for the observation that off-policy evaluation gives unreliable value estimates, which motivates using Q-functions produced by policy optimization."}],"review_version":1}