{"id":"20daa800-1005-417b-95ba-cf8e432316a0","arxiv_id":"2412.16848","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ACL-QL reports state-of-the-art D4RL results with per-transition adaptive conservatism in Q-learning, but its surrogate losses and CQL anchor are not rigorously justified.","lead":"ACL-QL is an offline reinforcement learning algorithm that learns a separate conservatism weight for each state-action pair instead of using one fixed level, and reports better D4RL benchmark scores than existing methods. The paper's theoretical derivation of the adaptive weights has gaps, and no code is released, so the empirical claims need independent confirmation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Surrogate losses (Eqs. 20-21) relax the conservative-level constraints instead of enforcing them, so the claimed Q-value interval guarantee does not hold for the implemented algorithm.","rationale":"The paper's headline contribution is a method that provably keeps the learned Q-function between the ordinary Q-function and CQL's Q-function by adaptively weighting each transition. For that guarantee to hold, the losses optimized in practice must enforce d_ord(s,a) >= 0 and d_cql(s,a) >= 0. Equations (20)-(21) are introduced as surrogate losses based on Lemma V.1, but the lemma gives ln x <= x-1, meaning 1+ln x <= x. Replacing a positive probability by a smaller quantity, which can also be negative, relaxes the inequality rather than preserving it. A concrete counterexample shows Eq. (20) can be zero while Eq. (16) is strictly positive, so the exact condition is violated without incurring any loss; the same applies to Eq. (21). Thus the bound claimed in Propositions IV.2 and IV.3 is not enforced by the implemented objective. The empirical evidence in Table VII and Figure 4 shows average or sampled Q-values in the intended direction, but it does not validate the pointwise interval guarantee. The CQL closed-form result is a secondary concern because it follows from the same idealized fixed-point argument and is less damaging than the surrogate-loss gap. Since the surrogate-loss issue is exactly the Reader's weakest assumption and remains unresolved, the appropriate disposition is unchanged: the paper is conditional pending either a correct derivation showing that Eqs. (20)-(21) imply the exact conditions, or an explicit retraction of the theoretical interval claim for the implemented algorithm. The proposed counterexample test would settle the logical gap immediately.","tokens_in":28943,"tokens_out":7205,"duration_ms":62099,"concrete_test":"Evaluate Eqs. (16) and (20) at mu=0.8, pi_beta=0.5, w_mu=1.1, w_pi_beta=1.0, d_ord=1.0. Eq. (20) evaluates to 0 while Eq. (16) evaluates to 0.12 > 0, showing the surrogate imposes no penalty even though the target condition is violated. To make the check exhaustive, grid over (mu, pi_beta) in (0,1)^2 and (w_mu, w_pi_beta, d_ord) in [0,5]^3 and count points where Eq. (20) is zero but Eq. (16) is positive; any such point settles the logical gap. For a behavioral check, re-run ACL-QL on a small toy MDP with exact losses (16)-(17) implemented in log-space compared to surrogate losses (20)-(21); if the exact version enforces the interval but the surrogate does not, the implemented algorithm's guarantee is void.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical claim is that ACL-QL keeps the learned Q-function between the ordinary Q-function and CQL's Q-function by enforcing d_ord(s,a) >= 0 and d_cql(s,a) >= 0 via the hinge losses in Eqs. (16)-(17). The implemented losses, Eqs. (20)-(21), replace the probabilities mu and pi_beta by 1+ln(mu) and 1+ln(pi_beta). Lemma V.1 only states ln x <= x-1, hence 1+ln x <= x, with equality only at x=1. The substituted quantities are therefore smaller than the original probabilities and can even be negative. Because the implication direction is not preserved, the surrogate is a relaxation, not an equivalent or a necessary condition: it can be zero while the exact conservative-level condition is violated. Concretely, with mu=0.8, pi_beta=0.5, w_mu=1.1, w_pi_beta=1.0, d_ord=1.0, Eq. (20) gives max(0, 1*(1+ln0.5) - 1.1*(1+ln0.8) + 1*(1+ln0.5)) = 0, whereas Eq. (16) gives max(0, 1*0.5 - 1.1*0.8 + 1*0.5) = 0.12 > 0. Thus minimizing Eq. (24) does not enforce d_ord(s,a) >= 0, and the same failure applies to Eq. (21) and d_cql. The theoretical guarantee in Propositions IV.2 and IV.3 is therefore disconnected from the actual objective. The empirical average Q-values in Table VII and Figure 4 are suggestive but only report aggregate or sampled behavior, not the pointwise interval bound. The CQL closed-form anchor is a secondary issue: it follows from the same idealized fixed-point argument and is less damaging than the surrogate-loss gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ACL-QL, an offline reinforcement learning algorithm that aims to keep the learned Q-function in a mild range between the ordinary Q-function and CQL's conservative Q-function. It introduces two learnable weight functions, w_mu and w_pi_beta, that modulate the conservative penalty per state-action pair, and it derives conditions on these weights under simplified assumptions. The practical algorithm uses transition-quality pseudo-labels, a monotonicity loss, and surrogate hinge losses to train the weights, Q-function, and policy. Experiments on D4RL Gym-MuJoCo, Adroit, and Franka Kitchen report state-of-the-art or competitive results, and ablations study the effect of each loss component and the transition-quality measurements.","tokens_in":29437,"tokens_out":7595,"duration_ms":62799,"significance":"The framework is well motivated, and the idea of per-transition conservative levels is a useful direction for reducing sensitivity to conservatism hyperparameters. The paper's main theoretical contribution, however, is not established: the implemented surrogate losses do not enforce the derived conditions, and the closed-form comparisons to CQL assume an idealized objective. If these gaps were closed, the method would offer a principled way to control conservatism in a fine-grained manner. The experimental study is extensive and includes a useful comparison against CQL with different alpha values, and the ablations are informative. The aggregate Q-value statistics in Table VII are suggestive but do not verify the pointwise 'mild range' claim. The flexibility of the proposed framework and the breadth of the D4RL evaluation are notable strengths.","major_comments":[{"comment":"The surrogate losses do not enforce the conditions in Eqs. (16)-(17). Lemma V.1 gives ln x <= x - 1, so 1 + ln x <= x, with equality only at x = 1; replacing mu and pi_beta by 1 + ln(mu) and 1 + ln(pi_beta) changes the inequality in an uncontrolled way because the same quantity appears with both positive and negative signs. Concretely, with mu = 0.8, pi_beta = 0.5, w_mu = 1.1, w_pi_beta = 1.0, and d_ord = 1.0, Eq. (20) equals 0 while Eq. (16) equals 0.12 > 0, so a zero surrogate loss does not imply the original condition; the same failure applies to Eq. (21) and d_cql. Since Eq. (24) is a sum of nonnegative hinge terms, the optimizer can drive these losses to zero even when the conservative-level conditions of Propositions IV.1 and IV.3 are violated. The theoretical guarantees are therefore not connected to the implemented objective, and the authors need either a stable version of the original probabilities or a proof of the intended implication.","section":"Section V-C, Eqs. (20)-(21)"},{"comment":"The closed-form expressions for the CQL Q-function and for d_cql assume a simplified objective that omits the max over mu, the regularizer R(mu), and the expectation over states; the proof in the appendix (Eq. (47)) starts from such a simplified problem. Consequently, the statement that Q^pi_CQL = Q^pi - alpha(mu - pi_beta)/pi_beta does not describe the actual CQL solution, and the pointwise interval Q_CQL <= Q_hat <= Q^pi is not established for the algorithms being compared. The authors should either state the precise idealizations under which Eqs. (5) and (10) hold or present them as design heuristics rather than formal guarantees.","section":"Section IV, Propositions IV.1 and IV.3"},{"comment":"The monotonicity loss in Eq. (15) enforces a relationship between softmax outputs over pairs in a batch, whereas Definition IV.1 specifies monotonicity with respect to the optimal Q-function for actions in the same state. The replacement of Q* by the transition-quality measurement m(s,a) and the use of cross-state pairs are not derived from the definition, so the loss is a heuristic. If the monotonicity property is central to the adaptive conservative-level mechanism, the authors should justify why the batch-level softmax equality approximates the per-state condition.","section":"Section V-B, Eq. (15)"},{"comment":"The empirical support for the 'mild range' claim reports only average Q-values over the dataset (Table VII) and a sampled visualization (Figure 4). An average can lie between the SAC and CQL averages even if the pointwise interval is violated for most transitions. To substantiate the claim that ACL-QL Q-values lie between the ordinary and CQL Q-values pointwise, the authors should report distributional evidence such as quantiles or per-transition Q differences.","section":"Section VI-B, Table VII and Figure 4"}],"minor_comments":[{"comment":"The heading reads 'Proof of Proposition VI.2' but should be 'Proof of Proposition IV.2'.","section":"Appendix, Proof of Proposition IV.2"},{"comment":"The denominator in the definition of err(s,a) is written as (1 - gamma sqrt(|D|)), which appears to be a typesetting error; the appendix Eq. (30) gives the correct form (1 - gamma) sqrt(|D|).","section":"Proposition IV.2, Eq. (9)"},{"comment":"The proof uses the symbol g(s,a) in Eqs. (38) and (46) where the main text defines h(s,a); please unify the notation.","section":"Proposition IV.2 and Appendix"},{"comment":"The baseline results are taken from different original papers, and standard deviations are not reported for the baselines, making it difficult to assess whether the reported improvements are statistically significant; this limitation should be acknowledged.","section":"Section VI-B, Table II"},{"comment":"The text states that the same hyperparameters are used for all tasks on the same environment, but alpha is set to 10, 20, or 1 depending on the environment; please clarify whether alpha is considered a per-environment hyperparameter.","section":"Section VI-A and Table I"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a revised arXiv version of a paper carrying an IEEE TNNLS DOI. The surrogate loss gap is a load-bearing issue that should be addressed before publication; the current theoretical analysis does not apply to the implemented algorithm. The empirical results are strong, but the central 'mild range' guarantee is not supported by the presented evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely new idea—per-transition adaptive conservative weights for CQL—and an extensive D4RL evaluation, but the theory as written doesn't connect to the implemented algorithm. The surrogate losses in Eqs. (20)-(21) don't enforce the claimed Q-value interval; they relax it. That's a load-bearing gap.\n\nWhat's new: ACL-QL replaces the fixed scalar alpha in CQL with two learned weight functions and adds a monotonicity loss that ties weights to transition quality. That's a clear departure from CQL, MQN, and mild-AC. The D4RL results are strong and, unusually, come from a single hyperparameter setting per environment. The ablations are thorough: they isolate each loss term, compare quality measurements, and check the discount factor. If the method holds up on replication, it's a useful advance for practitioners.\n\nWhere it breaks: the theory in Section IV derives pointwise Q-gap conditions under an idealized fixed-point CQL objective, ignoring the max and R(mu) terms. That's a standard simplification. The real problem is the conversion to losses. Eqs. (16)-(17) hinge on probabilities pi_beta and mu. Lemma V.1 gives ln x <= x-1, so 1+ln x <= x, and the substituted quantities can be negative. The surrogate losses are weaker than the original conditions: you can have the surrogate at zero while the exact condition is violated (the stress-test example with mu=0.8, pi_beta=0.5 shows this exactly). Minimizing Eq. (24) does not enforce d_ord >= 0 or d_cql >= 0, so the claimed 'between ordinary Q and CQL Q' range is not guaranteed for the actual objective. The empirical Q-values in Table VII are suggestive, but they are averages, not pointwise bounds.\n\nSecondary soft spots: the CQL anchor closed form in Prop. IV.3 is asserted from the same idealized fixed point, and the paper ships no code or baseline error bars. These are fixable in revision.\n\nBottom line: the empirical method may well work—the results are competitive—but the theoretical justification is disconnected from the training objective as written. That's a major revision, not a desk reject.\n\nFor you: worth a careful referee who checks the inequality direction and asks for either a proof that the surrogate preserves the conditions or a re-framed claim that doesn't rely on the interval guarantee. I'd send it to review.","headline":"A real idea and strong D4RL results, but the implemented surrogate losses relax the claimed interval guarantee instead of enforcing it.","tokens_in":29899,"tokens_out":2097,"would_cite":false,"duration_ms":19597,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ACL-QL argues that letting each state-action pair choose its own conservative level keeps offline Q-values in a mild range between the ordinary Q-function and CQL's conservative Q-function, and that this improves D4RL returns.","keywords":["offline reinforcement learning","Q-learning","conservative Q-learning","adaptive conservative level","distribution shift","D4RL benchmark","adaptive weight functions","monotonicity loss"],"falsifier":"Run ACL-QL on HalfCheetah-medium-v2 while logging, for every dataset transition, the argument inside the hinge losses in Eqs. (20) and (21); if a non-negligible fraction of transitions has the wrong sign, so the learned Q lies outside the claimed sandwich between ordinary Q and CQL Q, the central claim fails. A complementary check is to replace the logarithmic surrogate with the exact probabilities and see whether the reported performance gap vanishes.","tokens_in":28773,"feed_emoji":"🤖","tokens_out":7252,"duration_ms":60823,"temperature":0.7,"pith_summary":"Offline reinforcement learning has to stop Q-values from ballooning on unseen actions without squeezing them so much that the policy becomes useless. This paper proposes ACL-QL, which replaces the single fixed conservatism constant used by CQL with two learned weight functions, one for out-of-distribution actions and one for in-dataset actions. The central claim is that these per-transition weights keep the learned Q-function between the ordinary Q-function and CQL's conservative Q-function, and that this mild range is what lets one set of hyperparameters work across dataset types. On D4RL, ACL-QL reports total normalized returns of 241.6, 337.9, and 311.5 on HalfCheetah, Hopper, and Walker, above CQL's 199.7, 268.2, and 261.0.","feed_headline":"ACL-QL tunes conservatism per transition and tops D4RL","feed_subtitle":"Learned weights lift Q-values for good transitions, suppress bad ones, and improve returns on every dataset type.","key_machinery":"The engine is a pair of adaptive weight functions $w_\\mu(s,a)$ and $w_{\\pi_\\beta}(s,a)$, implemented as a single neural network with two outputs, trained against three objectives: a monotonicity loss that aligns weight differences with transition-quality differences, surrogate hinge losses that encode the two sandwich inequalities, and a positivity regularizer. The transition-quality pseudo-label $m(s,a)$ combines normalized Monte Carlo returns and immediate rewards for in-dataset pairs, and uses distance to the in-dataset action for OOD pairs. The paper also swaps probabilities for $1+\\ln\\mu$ and $1+\\ln\\pi_\\beta$ in the surrogate hinge losses to avoid arithmetic underflow, invoking $\\ln x \\le x-1$.","core_discovery":"The paper's discovery is that the conservative level of Q-learning can be made pointwise adaptive while remaining theoretically anchored. For any distribution $\\mu$ supported inside the behavioral policy, the gap between ACL-QL's Q-function and the ordinary Q-function is $(w_\\mu \\mu - w_{\\pi_\\beta}\\pi_\\beta)/\\pi_\\beta$, and the gap to CQL's Q-function is $((\\alpha-w_\\mu)\\mu - (\\alpha-w_{\\pi_\\beta})\\pi_\\beta)/\\pi_\\beta$. Forcing the first gap positive and the second positive sandwiches the learned Q-values between ordinary Q and CQL Q. ACL-QL sets the desired gaps from a normalized transition-quality score $m(s,a)$ and trains the weights with monotonicity, surrogate hinge, and positivity losses, with CQL as the special case where both weights equal the constant $\\alpha$.","pith_inferences":["A natural follow-up is to test whether the logarithmic surrogate, not the sandwich theory, is what drives the gains: replacing Eqs. (20)-(21) with the exact hinge losses (16)-(17) would separate the two effects, since the paper does not prove the surrogate preserves the inequalities.","The same sandwich construction could be layered onto other conservative offline algorithms that use a KL or distributional penalty, because the identities depend only on the CQL-style regularizer and not on the actor-critic backbone.","The transition-quality pseudo-label is a placeholder: upper-envelope estimates, learned uncertainty, or learned Q-based scores could replace it, and making $\\lambda$ in Eq. (13) depend on dataset quality might remove the last tuned hyperparameter."],"forward_implications":["If the sandwich claim holds, offline RL becomes less sensitive to the choice of conservatism hyperparameter; ACL-QL uses the same $\\alpha$ across tasks in one environment and still outperforms CQL at every fixed $\\alpha$ on the MuJoCo suite.","The learned Q-values stay in a mild range, so selecting a checkpoint by the dataset's average Q-value does not drift toward overestimation; the reported Q-values sit between SAC's ordinary Q-values and CQL-10's values.","Because CQL is the constant-weight special case, ACL-QL's improvements over CQL are direct consequences of letting conservative levels vary per transition rather than changing the underlying pessimism principle.","The framework allows custom weight functions beyond neural networks, so hand-designed rules could be substituted where learning weights is undesirable."],"supporting_citations":[{"why":"Supplies the base CQL objective, the fixed-$\\alpha$ conservative Q-function used as the lower anchor, and the special case where both ACL-QL weights equal $\\alpha$.","marker":"[18]"},{"why":"Defines the D4RL benchmark, the normalized score scale, and the dataset types on which ACL-QL reports its main results.","marker":"[70]"},{"why":"Provides the offline policy-evaluation procedure ACL-QL uses to select the final checkpoint by average dataset Q-value.","marker":"[76]"},{"why":"Serves as a leading baseline, especially on high-quality datasets such as Hopper-medium-expert, that ACL-QL claims to beat in sum.","marker":"[42]"},{"why":"A competing mild-conservatism offline actor-critic baseline whose MuJoCo results ACL-QL compares against.","marker":"[43]"},{"why":"A standard offline RL baseline that ACL-QL reports surpassing on the D4RL aggregate results.","marker":"[41]"}],"fun_headline_variants":["ACL-QL adapts Q-conservation per transition and tops D4RL","Per-transition Q-conservation: ACL-QL dominates D4RL","ACL-QL learns pointwise Q-conservatism, beating offline RL","ACL-QL: adaptive Q-conservation per state-action, D4RL winner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that replacing the action probabilities in the conservative-level losses by the logarithmic expressions $1+\\ln\\mu$ and $1+\\ln\\pi_\\beta$ leaves the claimed sandwich condition intact, even though those expressions can become negative.","fun_headline_variants_meta":{"raw":{"variants":["ACL-QL adapts Q-conservation per transition and tops D4RL","Per-transition Q-conservation: ACL-QL dominates D4RL","ACL-QL learns pointwise Q-conservatism, beating offline RL","ACL-QL: adaptive Q-conservation per state-action, D4RL winner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000869,"raw_usage":{"total_tokens":3790,"prompt_tokens":1000,"completion_tokens":2790,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":2702}},"tokens_in":616,"tokens_out":2790,"duration_ms":17992,"temperature":1.0,"reasoning_tokens":2702,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:16:03.081417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ACL-QL on HalfCheetah-medium-v2 while logging, for every dataset transition, the argument inside the hinge losses in Eqs. (20) and (21); if a non-negligible fraction of transitions has the wrong sign, so the learned Q lies outside the claimed sandwich between ordinary Q and CQL Q, the central claim fails. A complementary check is to replace the logarithmic surrogate with the exact probabilities and see whether the reported performance gap vanishes.","supporting_citations":[{"cited_title":"Conservative q-learning for offline reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the base CQL objective, the fixed-$\\alpha$ conservative Q-function used as the lower anchor, and the special case where both ACL-QL weights equal $\\alpha$."},{"cited_title":"Monotonic quantile network for worst-case offline reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Serves as a leading baseline, especially on high-quality datasets such as Hopper-medium-expert, that ACL-QL claims to beat in sum."},{"cited_title":"Mild policy evaluation for offline actor–critic,","cited_arxiv_id":null,"evidence_quote":"A competing mild-conservatism offline actor-critic baseline whose MuJoCo results ACL-QL compares against."},{"cited_title":"Offline reinforcement learning with implicit q-learning,","cited_arxiv_id":null,"evidence_quote":"A standard offline RL baseline that ACL-QL reports surpassing on the D4RL aggregate results."}],"review_version":1}