{"id":"9dfc073d-4534-4309-a791-2dd477072456","arxiv_id":"2501.04228","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CaR trains robots by maximizing a weighted sum of constraint functions with weights set by Lagrange multipliers, and it successfully taught a six-wheeled telescopic-legged robot to stand up from varied poses.","lead":"This paper proposes 'Constraints as Rewards' (CaR), a reinforcement learning method that replaces the reward function with a set of task constraints weighted by automatically learned Lagrange multipliers. The authors show the method trains a six-wheeled telescopic-legged robot, Tachyon 3, to stand up in simulation and on the real robot, where five manually designed reward functions failed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stated CaR objective Eq. (7) is unbounded in λ whenever any constraint slack is positive, so the formal basis for automatic weight tuning is invalid; Algorithm 1 actually descends on λ, solving a different saddle-point problem.","rationale":"The reader's conditional verdict already identifies the sign inconsistency between Eq. (5) and Algorithm 1 as load-bearing. My concern sharpens that: the deeper problem is that Eq. (7) is not merely mis-signed but mathematically unbounded, so no duality-gap or convergence argument can rescue it as written. The empirical claims are substantially supported: the paper includes a real-robot demonstration from multiple poses, an ablation showing each constraint's effect, a comparison against reward-based baselines, and an honest acknowledgment that the authors do not claim no reward function works. These are real evidence for the practical method. However, the central conceptual claim, that CaR automatically balances objectives through the Lagrange multipliers of Eq. (7), is invalidated by the ill-posed objective. Since the algorithm as implemented appears to be a standard primal-dual constrained RL method with zero reward, the paper needs a corrected derivation and a re-framing of what CaR actually optimizes. This is exactly a major-revision condition, not a rejection: the empirical result is credible, and the theory may be repairable by replacing the max over λ with the correct saddle-point formulation and analyzing that. The reader's CONDITIONAL verdict therefore stands unchanged.","tokens_in":11125,"tokens_out":6995,"duration_ms":77954,"concrete_test":"Construct a simple instance of the paper's Constraint 1 (Eq. 12) at a terminal state exactly at target, so g(T) = 10^-3 > 0, with r=0. Holding the policy fixed in Eq. (7), let the corresponding λ tend to infinity and note that the objective diverges linearly, proving no maximum over λ exists. Then instrument Algorithm 1 on the same setting and record the λ update: if λ decreases whenever G_m > 0, the implemented algorithm is solving max_π min_{λ≥0} L, not Eq. (7). This single check decides whether the stated optimization target is the one actually optimized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is that CaR, with r(s,a)=0, automatically balances objectives by solving max_{λ>0} max_π Σ_m λ_m E[Σ γ^t g_m(s,a)] (Eq. 7). This objective is not well-posed. For any policy with G_m(π) = Eπ[Σ γ^t g_m] > 0, the objective grows without bound as λ_m → ∞; for G_m(π) < 0, the supremum is approached only as λ_m → 0, which is excluded by the strict positivity constraint. Thus a finite maximizer over λ does not generically exist. This is more than a sign error in Eq. (5): the paper's own Algorithm 1 line 9 updates λ by gradient descent, i.e., it minimizes L over λ, which corresponds to the standard primal-dual saddle point max_π min_{λ≥0} L, not the max over λ in Eq. (7). Therefore the assertion that Lagrange multipliers act as automatically tuned weights among task objectives is not supported by the stated optimization. At best, the implemented algorithm solves a zero-reward constrained RL problem, and its behavior after all constraints are satisfied (λ → 0, loss of task gradient) requires a separate analysis. The empirical standing-up demonstration can stand independently, but the paper's theoretical foundation is unsound as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'Constraints as Rewards' (CaR), a method that replaces the reward function in a reinforcement learning problem with a set of constraint functions, and solves the resulting constrained problem with a Lagrangian-based algorithm called QRSAC-Lagrangian. The authors claim that solving the Lagrangian problem with a zero reward function automatically balances the weights of different task objectives, because the Lagrange multipliers act as the weights. The method is evaluated on a standing-up task for a six-wheeled telescopic-legged robot (Tachyon 3) in simulation and on the real robot, where it succeeds from all tested initial poses, whereas several manually designed reward functions fail. The paper also proposes four constraint-function designs and an ablation study of the five constraints used in the task.","tokens_in":11387,"tokens_out":3224,"duration_ms":32951,"significance":"If the theoretical foundation were sound, the contribution would be significant: it addresses reward engineering, a major practical bottleneck in robot learning, and the real-robot demonstration is a genuine asset. The paper ships a concrete algorithmic recipe (QRSAC-Lagrangian), a clear constraint-design taxonomy, and a reproducible-sounding experimental protocol. However, the central theoretical claim of automatic weight tuning rests on an optimization formulation that is ill-posed as written, and the actual algorithm solves a different saddle-point problem. The empirical standing-up result is credible and valuable, but the manuscript must be revised to make the optimization objective correct and to clarify what the Lagrange multipliers are doing. The current text does not support the strong statement that the weights are automatically balanced in the sense of Eq. (7).","major_comments":[{"comment":"The optimization problem in Eq. (7) is not well-posed. For any policy π with E_π[Σ γ^t g_m(s_t,a_t)] > 0, the objective Σ_m λ_m E_π[Σ γ^t g_m] grows without bound as λ_m → ∞, so no finite maximizer over λ exists; for E_π[Σ γ^t g_m] < 0, the supremum is approached only as λ_m → 0, which is excluded by the strict constraint λ_m > 0. Thus the 'max over λ' in Eqs. (5) and (7) cannot be the basis for the claimed automatic weight tuning.","section":"Section IV-A, Eq. (5) and Eq. (7)"},{"comment":"Algorithm 1 updates λ by gradient descent (λ_m ← λ_m − Adam(α_λ, ∇_{λ_m} L(π, λ))) and clamps to nonnegative values. This is the standard saddle-point update for min_{λ≥0} max_π L(π,λ), not the max over λ of Eq. (7). The manuscript therefore describes and analyzes one objective while implementing another. The theoretical justification for automatic balancing collapses unless the text is rewritten to state the actual saddle-point problem and to explain how the zero-reward limit fits within it.","section":"Algorithm 1, line 9"},{"comment":"The claim that 'Lagrange multipliers, which serves as weights among constraints, can be tuned automatically' is only true in a limited sense. In the standard constrained RL interpretation, λ_m increases for violated constraints and decreases toward zero for satisfied constraints. In the zero-reward case, once all constraints are satisfied, the Lagrangian objective becomes identically zero and provides no further learning signal; the policy is then only kept inside the feasible set by the constraints themselves. The paper should state this mechanism explicitly rather than implying that Eq. (7) yields a well-posed weighted-sum maximization.","section":"Section IV-A, paragraph after Eq. (7)"},{"comment":"The learning curve shows the average of 5 runs but no error bars or variance shading, and Table III reports averages of 10 runs without variance. The claims that QRSAC-Lagrangian converges 'faster and more robustly' than SAC-Lagrangian, and that each ablation constraint improves its corresponding metric, need uncertainty quantification; as presented, the differences in Table III (e.g., Pose 0.925 vs 0.920 vs 0.901) may be within run-to-run noise.","section":"Section VI-B, Fig. 4a and Table III"}],"minor_comments":[{"comment":"The text calls Eq. (5) the 'Lagrange dual function' and its maximization the 'Lagrange dual'; in standard constrained optimization, the dual problem for constraints E[Σ γ^t g_m] ≥ 0 is min_{λ≥0} max_π L(π,λ), not max_{λ>0} max_π L(π,λ). Please correct the terminology.","section":"Section IV-A, Eq. (5)"},{"comment":"Algorithm 1 initializes λ = 0 and clamps to max(λ_m, 0), which is inconsistent with the strict positivity constraint λ_m > 0 in Eq. (5). The paper should use λ ≥ 0 consistently.","section":"Algorithm 1 and Section IV-A"},{"comment":"The caption says 'The weight of the pose parameter shown on the right is the weight of the left-front hip joint', but the figure shows the weight of a pose constraint; please clarify the wording.","section":"Section VI-B, caption of Fig. 4"},{"comment":"Typo: 'sustituting' should be 'substituting'.","section":"Appendix I-D"},{"comment":"Typo: 'distrubution' should be 'distribution'.","section":"Section VI-B"},{"comment":"Typo: 'accelaration' should be 'acceleration'.","section":"Table VI"}],"recommendation":"major_revision","confidential_remarks":"The paper's core experimental result — that a policy trained with CaR stands the robot up from all tested poses in simulation and on the real robot — appears sound and is a valuable contribution. The main obstacle is the theoretical framing: Eq. (7) as written is genuinely ill-posed, and the algorithm descends on λ, so the 'automatic balancing' story needs to be rewritten around the saddle-point formulation. This is fixable without changing the experiments. I would also gently push the authors to soften the 'without reward functions' phrasing in the title/abstract, since the Lagrangian objective is still a weighted sum of (constraint) functions and the method is best understood as constrained RL with a zero extrinsic reward; the overclaim could generate unnecessary controversy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an empirical paper with a real robotic result, but the theoretical claim as written doesn't hold up. Equation (7) is not a well-posed optimization problem, and Algorithm 1 solves something different from what the equations say.\n\nThe genuine contributions: a novel six-wheeled telescopic-leg robot (Tachyon 3) standing up from arbitrary poses, with the policy transferring to the real robot without fine-tuning; four clean constraint templates that are useful for task specification; and the QRSAC-Lagrangian combination, which does seem to stabilize learning when target Q distributions change. The ablation study is honest and shows each constraint contributes. The authors also correctly note that CaR is limited to constraint-specifiable tasks.\n\nThe soft spots are serious. The core equation (7), max over λ and π of Σ λ_m E[G_m], is unbounded whenever any expected constraint value is positive, so no finite maximizer exists. The paper calls it a Lagrange dual, but the dual is a minimization over λ. Algorithm 1 line 9 updates λ by gradient descent, i.e., it minimizes L over λ—the standard primal-dual heuristic. That means the implemented algorithm is not optimizing the stated objective. Consequently, the central 'automatic balancing of weights' claim is not supported by the math. What the algorithm actually does is a zero-reward constrained RL problem, and whether the multipliers converge or even stay bounded after constraints are nearly satisfied requires a separate analysis. This is more than a sign typo; the whole Lagrangian formulation with r=0 is degenerate.\n\nThe empirical presentation also needs work: no error bars on the learning curves, no variance in the ablation table, and no code or data release. The comparison to manually designed rewards is fair but limited—five hand-picked reward functions.\n\nBottom line: the robot result is valuable and the constraint templates are useful, but the paper's formal foundation is incorrect as written. A serious referee should ask for a corrected derivation or a reframing of CaR as a heuristic procedure rather than a principled optimization, plus proper statistics and artifacts. I'd send it to review with major revision, but I would not cite it until the theory is fixed.","headline":"Real robot result, but the theory as written is wrong: Eq (7) is unbounded and Algorithm 1 minimizes over λ rather than maximizing, so the automatic-weighting claim doesn't follow.","tokens_in":11958,"tokens_out":5659,"would_cite":false,"duration_ms":53663,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Setting the reward to zero and expressing the task as constraints lets Lagrange multipliers balance objectives automatically, and a six-wheeled robot learns to stand up where hand-designed reward functions fail.","keywords":["constraints as rewards","reward engineering","reinforcement learning with constraints","Lagrangian method","QRSAC","legged robot standing up","Tachyon 3","multi-objective balancing"],"falsifier":"Run the same training while logging each constraint weight $\\lambda_m$ and the Lagrangian value $L$; if the multiplier update lowers $L$ whenever a constraint is violated, then the update is moving opposite to the max-over-$\\lambda$ objective in Eq. (7), and the stated automatic-balancing justification is not what the algorithm implements.","tokens_in":10908,"feed_emoji":"🤖","tokens_out":9104,"duration_ms":81876,"temperature":0.7,"pith_summary":"Constraints as Rewards (CaR) is a proposal for training robots without writing a reward function: the task is described by constraint functions, the reward is set to zero, and the Lagrange multipliers serve as the weights that balance the task objectives. The paper argues this removes the trial-and-error of reward engineering because the weights adjust automatically during training instead of being tuned by hand. To make the idea practical, it introduces QRSAC-Lagrangian, a quantile-based actor-critic that alternates policy updates with multiplier updates, along with four reusable constraint templates covering timestep and episode requirements. It applies the method to the standing-up task of a six-wheeled telescopic-legged robot, Tachyon 3, and reports that the learned policy stands the robot up in simulation, on unfamiliar terrain, and on the physical robot, while several manually designed reward functions fail to learn the task. The paper's central claim is that, for tasks expressible as constraints, reward engineering can be replaced by constraint design with automatically balanced weights.","feed_headline":"No reward function: constraints alone teach a robot to stand up","feed_subtitle":"Constraint multipliers tune themselves as the six-legged Tachyon 3 learns to rise.","key_machinery":"The load-bearing object is the Lagrangian objective with zero reward, $L(\\pi, \\lambda) = \\sum_{m=1}^{M}\\lambda_m \\mathbb{E}_\\pi\\left[\\sum_{t=0}^{T}\\gamma^t g_m(s_t,a_t)\\right]$, maximized with respect to both the policy and the multiplier vector; the multipliers are the weights that balance task objectives. Carrying the practical implementation is QRSAC-Lagrangian, an extension of QRSAC whose quantile critic estimates the distribution of Q values rather than a point estimate, which the authors argue is more stable when the target distribution shifts as the multipliers update. The four constraint templates are timestep probability constraints, timestep value constraints, episode probability constraints, and episode value constraints, each written so that the requirement $\\mathbb{E}\\left[\\sum_t \\gamma^t g(s_t,a_t)\\right] \\ge 0$ has an intuitive interpretation in terms of the task.","core_discovery":"The paper's central claim is that reward engineering can be sidestepped by replacing the reward function with a set of constraint functions and setting the reward to zero. The resulting objective, $\\max_{\\lambda_m > 0, \\pi} \\sum_{m=1}^{M} \\lambda_m \\mathbb{E}_\\pi\\left[\\sum_{t=0}^{T}\\gamma^t g_m(s_t,a_t)\\right]$, is a Lagrange dual form of a constrained reinforcement learning problem, and the multipliers act as automatically tuned weights among the objectives. To solve this objective, the paper proposes QRSAC-Lagrangian, an extension of QRSAC whose quantile critic estimates the distribution of Q values rather than a point estimate, arguing that this copes with the shifting distribution of target values as multipliers update. In experiments on Tachyon 3, the method is reported to learn the standing-up motion from arbitrary initial poses in simulation, generalize to rough and sloped terrain, and transfer to the physical robot, while five manually designed reward functions and two of the three comparison algorithms fail to learn the task.","pith_inferences":["If the pseudocode is taken literally, the multiplier update descends rather than ascends, so the implemented objective may be the conventional constrained-RL dual (min over $\\lambda$ of the max over $\\pi$) rather than the max-over-$\\lambda$ objective in Eq. (7); the reported balancing would then be an empirical feature of the alternating scheme, not a consequence of the stated dual problem.","A direct test: freeze the multipliers at their final learned values and retrain from scratch. If the final policy is unchanged, the dynamic adjustment is not what produces the behavior; if it degrades, the adaptive weighting is load-bearing.","A natural extension is a hybrid objective that adds a small reward for pure maximization while keeping constraints for safety and posture, since the paper's own limitation discussion points toward combining rewards with constraints.","The quantile critic's apparent robustness to shifting target distributions could transfer to other reinforcement learning settings with non-stationary objectives, such as curriculum learning or meta-learned rewards."],"forward_implications":["For tasks expressible as constraints, designers can skip hand-tuning reward weights; the multiplier updates supply the balance during training.","QRSAC-Lagrangian is reported to converge faster and more stably than SAC-Lagrangian, PPO-Lagrangian, and CaT on the standing-up task and on an inverted pendulum benchmark.","A policy trained only in a flat simulation stands Tachyon 3 up on rough terrain, a 10-degree slope, and the physical robot without fine-tuning.","CaR does not remove objective design; it moves the burden from weighted reward functions to inequality constraints, and it leaves pure maximization tasks such as 'walk as fast as possible' outside its scope.","The four constraint templates give task designers a reusable vocabulary for composing terminal and throughout-episode requirements."],"supporting_citations":[{"why":"Provides the Lagrange dual formulation that CaR uses to convert the constrained problem into an unconstrained objective.","marker":"[6]"},{"why":"Defines QRSAC, the quantile-critic reinforcement learning algorithm that QRSAC-Lagrangian extends.","marker":"[7]"},{"why":"Describes CaT, a constraints-as-terminations baseline that the paper compares against and which fails to learn the standing-up task.","marker":"[4]"},{"why":"Supplies the PPO-Lagrangian baseline used in the learning-curve comparison.","marker":"[8]"},{"why":"Supplies the SAC-Lagrangian baseline, which reaches similar final performance but converges more slowly.","marker":"[9]"},{"why":"Documents the Tachyon 3 robot's hardware and joint configuration, defining the task domain.","marker":"[10]"},{"why":"Describes earlier standing-up learning with episodic rewards, motivating the paper's final-pose constraint design.","marker":"[5]"},{"why":"Provides the physics engine used to simulate the robot for training and evaluation.","marker":"[21]"}],"fun_headline_variants":["Constraints replace rewards: robot learns to stand without reward tuning","Reward-free RL: constraints auto-balance as robot learns to rise","No reward function: constraints auto-tune as robot rises","No reward engineering: constraints alone enable robot to stand"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that repeatedly alternating between improving the policy and adjusting the constraint weights converges to the balanced solution the constrained problem asks for, even though the optimization problem is not convex and the weight update in the pseudocode moves opposite to the direction the equations prescribe.","fun_headline_variants_meta":{"raw":{"variants":["Constraints replace rewards: robot learns to stand without reward tuning","Reward-free RL: constraints auto-balance as robot learns to rise","No reward function: constraints auto-tune as robot rises","No reward engineering: constraints alone enable robot to stand"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001073,"raw_usage":{"total_tokens":4491,"prompt_tokens":944,"completion_tokens":3547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":3478}},"tokens_in":560,"tokens_out":3547,"duration_ms":23726,"temperature":1.0,"reasoning_tokens":3478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:39:41.967282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same training while logging each constraint weight $\\lambda_m$ and the Lagrangian value $L$; if the multiplier update lowers $L$ whenever a constraint is violated, then the update is moving opposite to the max-over-$\\lambda$ objective in Eq. (7), and the stated automatic-balancing justification is not what the algorithm implements.","supporting_citations":[{"cited_title":"Outracing champion gran turismo drivers with deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Defines QRSAC, the quantile-critic reinforcement learning algorithm that QRSAC-Lagrangian extends."},{"cited_title":"Cat: Constraints as terminations for legged locomotion reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Describes CaT, a constraints-as-terminations baseline that the paper compares against and which fails to learn the standing-up task."},{"cited_title":"Benchmarking safe exploration in deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the PPO-Lagrangian baseline used in the learning-curve comparison."},{"cited_title":"Learning to walk in the real world with minimal human effort,","cited_arxiv_id":null,"evidence_quote":"Supplies the SAC-Lagrangian baseline, which reaches similar final performance but converges more slowly."},{"cited_title":"Real-time perceptive motion control using control barrier functions with analytical smoothing for six-wheeled- telescopic-legged robot tachyon 3,","cited_arxiv_id":null,"evidence_quote":"Documents the Tachyon 3 robot's hardware and joint configuration, defining the task domain."},{"cited_title":"Learning agile and dynamic motor skills for legged robots,","cited_arxiv_id":null,"evidence_quote":"Describes earlier standing-up learning with episodic rewards, motivating the paper's final-pose constraint design."},{"cited_title":"Mujoco: A physics engine for model-based control","cited_arxiv_id":null,"evidence_quote":"Provides the physics engine used to simulate the robot for training and evaluation."}],"review_version":1}