{"id":"555f877e-f0a0-44de-847c-0aea4004d0c3","arxiv_id":"2607.04854","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CARL improves LLM planning by rewarding the KL divergence between output distributions on constrained versus unconstrained queries, yielding large gains on BlocksWorld, TravelPlanner, and T-Eval.","lead":"CARL trains LLMs for planning by adding a reward that measures how much the model's output distribution shifts when constraints are removed from the input. This makes small models produce far more constraint-compliant plans than standard RL fine-tuning or large reasoning models, without external solvers.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The KL reward can be gamed by superficial distributional shifts that do not improve plan validity, and the paper never isolates whether RCA is causally responsible for the reported gains.","rationale":"The reader correctly isolates the non-hackability of Eq. 6 as the weakest assumption. The paper supplies strong correlational evidence (attribution shift, ablations of masking and discrepancy metrics, multi-algorithm transfer) but never a causal isolation of RCA. Because the same policy produces both sides of the KL, superficial gaming remains possible and untested. The concrete control above would settle the issue with a single set of three short GRPO runs; until it is run, the central claim stays conditional rather than fully supported. No stronger internal inconsistency is present, so the verdict remains CONDITIONAL.","tokens_in":23658,"tokens_out":581,"duration_ms":5549,"concrete_test":"On the TravelPlanner validation set, retrain Qwen3-8B under three GRPO variants that keep Rtask identical: (1) true CARL (Eq. 7), (2) RCA replaced by a length-normalized KL between two independent unconstrained rollouts (null divergence), (3) RCA replaced by a random continuous reward of matching mean/variance. If final pass rates of (2) and (3) remain within 3 points of (1), the KL-as-constraint-awareness assumption is falsified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on RCA(x,y)=DKL[π\theta(y|x)∥π\theta(y|x\\C)] (Eq. 6) being a faithful, non-hackable proxy for genuine constraint awareness that is then added into the GRPO objective (Eq. 7). Because the same policy generates both the constrained and unconstrained distributions, the model can increase the KL by any systematic change in token probabilities that is correlated with the mere presence of constraint tokens, without those changes improving the validity of the plan. The paper never reports a control that severs this link: there is no experiment that (a) freezes or randomizes RCA while keeping Rtask, (b) substitutes a non-semantic divergence (e.g., length or entropy difference), or (c) measures whether high-RCA rollouts that still fail the task validator are down-weighted. Attribution scores (Fig. 5) and the reward-masking ablation (Table 3) are only correlational; they do not prove that the KL term is what drives the +11.1 % final-pass-rate jump on TravelPlanner. If the observed gains are largely produced by the extra forward pass or by the mere presence of an auxiliary continuous signal, the strongest claim does not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces CARL, a reinforcement learning framework for LLM planning that decomposes the reward into a binary task success term R_task and a continuous constraint-aware term R_CA = D_KL[π_θ(y|x) || π_θ(y|x\\C)], where x\\C is the input with constraint tokens removed (Eqs. 4–6). This is plugged into GRPO (Eq. 7), PPO, and a preference-construction variant of DPO. The claim is that maximizing the distributional shift induced by constraints strengthens the model’s intrinsic focus on them, yielding higher plan validity without external solvers or larger models. Empirical support comes from BlocksWorld, TravelPlanner (final pass rate 56.1 % for Qwen3-8B vs. 45 % RFT and 12.2 % DeepSeek-R1), and T-Eval, plus ablations on masking targets, distance metrics, α, model scale, training dynamics, attribution scores, and limited cross-domain transfer.","tokens_in":24022,"tokens_out":1127,"duration_ms":30346,"significance":"If the reported gains are robust, CARL supplies a practical, model-agnostic route to more reliable constrained planning that stays inside a single lightweight policy and avoids the engineering overhead of neuro-symbolic or multi-agent scaffolds. The multi-benchmark, multi-algorithm, multi-scale evaluation, the explicit efficiency comparison (Table 5), the attribution analysis (Fig. 5), and the observation that the continuous signal aids exploration for weak base models are concrete strengths that raise the work above a pure engineering tweak. The approach is therefore of clear interest to the planning-with-LLMs community and to anyone seeking intrinsic rather than scaffolded competence.","major_comments":[{"comment":"Section 4.2, Eqs. (6)–(7) and the GRPO objective: the central causal claim is that R_CA is a faithful, non-hackable proxy for genuine constraint awareness. Table 3 (masking goals vs. constraints) and the stable dynamics in Fig. 7 / App. B are supportive, yet they do not fully isolate the semantic content of the KL from generic effects of an auxiliary continuous reward or the extra forward pass. A control that (a) freezes or randomizes R_CA while retaining R_task, (b) replaces the constraint ablation with a non-semantic one (random tokens or length/entropy difference), or (c) reports the correlation of R_CA with validity among failed rollouts would more rigorously establish that the +11 pp final-pass jump on TravelPlanner is driven by constraint focus rather than by the mere presence of a smooth auxiliary signal. Without such a control the strongest interpretation remains under-supported.","section":"§4.2, Eqs. (6)–(7), Table 3"},{"comment":"Main results (Table 1) and implementation details: all headline numbers are reported from single runs with no error bars, confidence intervals, or multi-seed statistics, despite very small training sets (100 / 45 / 128 examples). Given that the key claim is a substantial absolute gain over RFT (e.g., 56.1 % vs. 45.0 % final pass on TravelPlanner), the absence of variance estimates makes it difficult to judge whether the improvements are stable or could be explained by seed or hyper-parameter sensitivity. At minimum, 3–5 independent seeds with standard deviations should be added for the primary GRPO comparisons.","section":"Table 1, §5.1"}],"minor_comments":[{"comment":"Clarify the precise placement of the sequence-level R_CA inside the token-averaged GRPO objective (Eq. 7): is the KL simply added once per sequence and then normalized by |y_i|, or is it broadcast to every token before averaging? A one-sentence statement would remove ambiguity.","section":"Eq. (7)"},{"comment":"Figure 5 (attribution) and the case studies in App. H would be easier to interpret if the color scale and the exact Captum FeatureAblation settings (baseline, aggregation) were stated in the caption or main text.","section":"Fig. 5, App. H"},{"comment":"The hyper-parameter α = 0.001 is used throughout the main experiments; the sensitivity plot (Fig. 8) is helpful but appears only in the appendix. A brief forward reference in §5.1 would improve readability.","section":"§5.1, App. F"},{"comment":"Minor typographical inconsistencies: “non self-driving” vs. “non-self-driving”, occasional missing spaces around mathematical operators, and the arXiv identifier formatting in the header.","section":"throughout"}],"recommendation":"minor_revision","confidential_remarks":"The core idea is clean and the empirical package is already stronger than many concurrent LLM-planning papers. The missing causal control and lack of error bars are the only load-bearing gaps; both are fixable within a revision cycle. I see no novelty or citation-pattern issues that would require editorial attention beyond the usual."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing is that they take the KL between the policy’s own outputs under constrained vs. unconstrained inputs and add it as a continuous auxiliary reward. That single change, plugged into GRPO/PPO/DPO, lifts final pass rates a lot—most visibly Qwen3-8B on TravelPlanner from 2.2 % to 56.1 %, well above R1 and o1-preview zero-shot. The idea is not in the cited RLHF or planning literature, and they show it works across model sizes, three RL algorithms, and both explicit and implicit constraints.\n\nWhat they do well: the reward is cleanly separated from the binary task indicator, the masking ablation (Table 3) shows that only constraint ablation helps while goal ablation hurts, the distance-metric ablation (Table 4) rules out MSE collapse, and the attribution shift (Fig. 5) plus case studies line up with the claim of increased constraint focus. Training dynamics stay stable under the usual KL penalty, and the extra forward pass is cheap relative to simply doubling rollouts. Generalization checks to ALFWorld/WebShop/TripCraft are modest but positive, which is more than most planning papers bother with.\n\nSoft spots, in proportion. The stress-test worry that the model can game the KL with superficial token shifts is real; they never freeze or randomize RCA while keeping Rtask, so we cannot fully isolate causality. That is a genuine gap, not a fatal one—the reward-masking and multi-algorithm consistency still make the gains hard to dismiss as pure noise. Constraint extraction on T-Eval leans on GPT-4o, and no code/data are released, both addressable. Free parameters (α, β, group size) are few and they show a reasonable sensitivity range.\n\nThis is for people who actually train planning agents and care about intrinsic constraint compliance without solvers. The math is standard, the data are public benchmarks, the citations are appropriate. I would send it to peer review; the core technique is solid enough that referees can demand the missing control experiments. Worth a look if you are working on agent reliability.","headline":"Clean, algorithm-agnostic RL reward that measurably raises constraint compliance on three planning benchmarks; the KL-proxy assumption is the softest point but the ablations and multi-algorithm results still make the paper worth reading.","tokens_in":24607,"tokens_out":534,"would_cite":true,"duration_ms":5846,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A KL-based reward that forces language models to notice constraints while they plan raises final plan validity far above ordinary RL fine-tuning.","keywords":["constraint-aware RL","LLM planning","KL divergence reward","TravelPlanner","BlocksWorld","GRPO","intrinsic constraint focus"],"falsifier":"Train two otherwise identical models, one with and one without the KL term; if the KL-trained model shows higher average KL yet no higher final-pass rate (or lower attribution scores on constraint tokens) on held-out multi-constraint queries, the reward is misaligned and the central claim fails.","tokens_in":24585,"feed_emoji":"🧠","tokens_out":809,"duration_ms":6868,"temperature":0.7,"texified_at":"2026-08-05T21:15:07.471176+00:00","pith_summary":"Language models can reason and recall facts, yet they routinely produce plans that ignore explicit constraints such as budgets, dietary rules or physical preconditions. The authors argue that the missing ingredient is an intrinsic, continuous signal that tells the model how much its own output distribution actually depends on those constraints. They introduce CARL: a reinforcement-learning objective that adds the KL divergence between the model’s predictions under the full constrained query and under the same query with constraints stripped away. Maximizing that divergence, together with ordinary task success, teaches the model to keep constraints in its generation process rather than off-loading them to external solvers. On BlocksWorld, TravelPlanner and tool-use planning the resulting models outperform both standard RL fine-tuning and much larger zero-shot reasoners, while attribution scores confirm that constraint tokens receive systematically higher attention.","texify_model":"deepseek-v4-flash","texify_usage":{"total_tokens":2216,"prompt_tokens":535,"completion_tokens":1681,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1235}},"feed_headline":"KL reward makes LLMs obey constraints while planning","feed_subtitle":"Final pass rates jump past ordinary RL and larger zero-shot reasoners without external solvers","key_machinery":"The constraint-aware reward RCA: the KL divergence between the model’s output distribution given the full query and the same distribution given the query with constraint tokens removed; maximizing it forces the policy to treat constraints as indispensable.","core_discovery":"Adding the continuous reward $R_{CA}(x,y)=D_{KL}[\\pi_\\theta(y|x) \\| \\pi_\\theta(y|x\\setminus C)]$ to standard on-policy or off-policy RL objectives produces language models whose final pass rates on multi-constraint planning benchmarks substantially exceed both ordinary reinforcement fine-tuning and zero-shot state-of-the-art reasoners, because the models learn to keep constraints inside their own generation process rather than neglecting them.","pith_inferences":["Any generation task whose inputs can be cleanly split into “goal” and “side conditions” (safety rules, style guides, resource limits) is a candidate for the same KL reward.","If the divergence can be computed on partial prefixes, the method may stabilize multi-turn interactive agents that must obey long-horizon constraints.","The continuous nature of the reward suggests it could regularize other preference-optimization methods that currently rely only on binary success labels."],"forward_implications":["Medium-size open models can be turned into reliable planners without external solvers or larger teacher models.","The same KL reward can be dropped into PPO, GRPO or DPO pipelines with only a second forward pass.","Attribution scores on constraint tokens become a diagnostic that tracks planning success.","Constraint-aware fine-tuning generalizes from symbolic blocks to real-world travel and tool-use domains."],"fun_headline_variants":["KL reward forces LLMs to heed constraints in planning","Constraint-aware RL via KL keeps plans valid without solvers","RCA KL term embeds constraint focus into LLM generation","On-policy KL reward lifts LLM pass rates on multi-constraint tasks","CARL trains LLMs to retain constraints inside the generation loop"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That a model which increases the KL divergence between constrained and unconstrained outputs has truly become more constraint-aware, rather than merely gaming the continuous reward with superficial token-level changes that leave plan validity unchanged.","fun_headline_variants_meta":{"raw":{"variants":["KL reward forces LLMs to heed constraints in planning","Constraint-aware RL via KL keeps plans valid without solvers","RCA KL term embeds constraint focus into LLM generation","On-policy KL reward lifts LLM pass rates on multi-constraint tasks","CARL trains LLMs to retain constraints inside the generation loop"]},"model":"grok-4.5","effort":"low","cost_usd":0.00359,"raw_usage":{"total_tokens":1161,"prompt_tokens":748,"num_sources_used":0,"completion_tokens":85,"cost_in_usd_ticks":35900000,"prompt_tokens_details":{"text_tokens":748,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":328,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":748,"tokens_out":85,"duration_ms":3183,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T12:28:08.116182+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train two otherwise identical models, one with and one without the KL term; if the KL-trained model shows higher average KL yet no higher final-pass rate (or lower attribution scores on constraint tokens) on held-out multi-constraint queries, the reward is misaligned and the central claim fails.","supporting_citations":[],"review_version":1}