{"id":"d627f364-2059-401c-9db8-b040136dbb92","arxiv_id":"2501.12539","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CERLLA uses a GPT-4 semantic parser that learns from environment rollouts to map language instructions to Boolean compositions of pretrained value functions, reaching 92% success on 162 BabyAI tasks with lower sample complexity than a non-compositional baseline.","lead":"The paper builds a system called CERLLA that uses a large language model to translate instructions into symbolic expressions, which then compose pretrained policies to solve 162 gridworld tasks. It reports that this composed method reaches an oracle-level 92 percent success rate with fewer environment steps than a non-compositional baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing ablation: CERLLA's sample-complexity advantage could come from GPT-4's few-shot parsing plus compositional value functions, not from the environment-rollout feedback the paper claims as its central novelty.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption identified as the held-out generalization relying on GPT-4's ability to parse novel instructions from retrieved examples. I agree that this is a real dependence, and the paper's GPT-3.5 results show the failure mode is possible. However, I see an even more direct threat to the central claim: the paper never demonstrates that its titular learning mechanism--environment-rollout feedback shaping the in-context examples--is causally responsible for the reported sample-complexity advantage. The entire learning loop is a trial-and-error filter over GPT-4's sampled expressions; without a control where the examples are fixed and not learned, the observed performance could be entirely due to (a) the pretrained compositional value functions and (b) GPT-4's few-shot ability to map language to Boolean expressions once a few random symbol bindings are shown. The paper's contribution list explicitly claims novelty for learning a parser from rollout feedback, and the abstract says the parser is 'trained using reinforcement learning.' If a static prompt achieves the same result, those statements are overstated and the method reduces to composing value functions with a few hand-provided (or oracle) examples. This concern is testable with a simple ablation, and until it is run, the central claim that the RL feedback loop drives the improvement is unverified. The reader did mention 'missing ablations, especially a static-prompt baseline without rollout feedback' in the rationale, so there is partial agreement, but the reader's primary weakest assumption is about GPT-4 generalization. I recommend keeping the verdict CONDITIONAL: the paper is plausible and well-motivated, but this missing control is essential before the methodological contribution can be accepted. The compute asymmetry between the 162-network WVF ensemble and the single-network baseline is also worth noting, but it is secondary because the paper's metric is sample complexity and it does penalize pretraining steps; the static-prompt ablation directly targets the claimed mechanism.","tokens_in":12290,"tokens_out":6482,"duration_ms":70183,"concrete_test":"Run the identical 162-task learning experiment with a static-prompt variant of CERLLA: replace the learned in-context example set with a fixed set of 10 ground-truth (instruction, expression) pairs chosen to cover all nine symbols and the three operators (e.g., taken from the Oracle agent's mappings or from CERLLA's first successful examples), disable all environment-rollout feedback, and keep the same BM25 retrieval, beam width, and evaluation protocol. If the static variant reaches the same ~92% success rate within the same total environment steps (or close to it), the paper's claim that the parser is 'trained using RL' is not supported and the central novelty should be reframed. If the static variant fails or is substantially worse, the environment-rollout feedback is demonstrated to be load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central methodological contribution is a semantic parser 'trained using reinforcement learning and in-context learning' (Abstract) and 'the first to learn a semantic parser using only in-context learning with feedback from environment rollouts' (Contribution 2). Yet the only learning mechanism is a verifier: sample 10 candidate Boolean expressions from GPT-4, roll each out for 100 episodes, and retain the expression if success >=92%. This accumulates in-context examples that are later retrieved via BM25. No experiment isolates whether this feedback loop causes the observed performance. In particular, a static-prompt baseline--where a fixed set of grounded (instruction, expression) examples is provided without any environment feedback or refinement--is not run. If such a static prompt already yields ~92% success on the 162 tasks, then the 'RL training' of the parser is causally irrelevant; the result would be fully explained by the pretrained compositional value functions plus GPT-4's few-shot ability to infer the random symbol-to-attribute mapping from a handful of examples. The paper's own GPT-3.5 results (Appendix Table 3) show the parser can accumulate 'inconsistent or needlessly complicated' expressions, suggesting the learned example set is noisy, but without an ablation the contribution of the rollout signal to the GPT-4 results remains unquantified. This concern is load-bearing because the headline claim--sample-complexity advantage over a non-compositional baseline--does not distinguish the full CERLLA pipeline from a simpler one that omits the learning loop entirely.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CERLLA, a method that combines pretrained compositional world value functions (WVFs) with an LLM-based semantic parser for language-conditioned RL tasks. In a first phase, nine attribute-level WVFs are pretrained, each implemented with 18 CNN-DQN heads. In a second phase, GPT-4 (or GPT-3.5) is prompted with up to 10 BM25-retrieved in-context examples to produce 10 candidate Boolean expressions for a natural-language mission; each candidate is instantiated as a composed WVF and rolled out for 100 episodes, and expressions with success rate at least 92% are added to the in-context example set. The paper claims that CERLLA solves 162 simultaneous BabyAI tasks with a 92% success rate, matching an oracle upper bound, while a non-compositional baseline reaches only about 80% after 21 million steps, and that CERLLA generalizes to held-out tasks. The central assertion is that the environment-rollout feedback loop, together with compositional value functions, yields a sample-complexity advantage over learning a single joint policy.","tokens_in":12530,"tokens_out":3850,"duration_ms":41903,"significance":"If the central claims hold, the paper makes a useful contribution by connecting compositional value-function algebra with in-context LLM parsing, and it reports a large-scale (162-task) empirical study with an interpretable decomposition. The paper is honest about the GPT-3.5 failure mode and includes a limitations section. However, the evidence as presented does not yet establish the load-bearing causal claim that environment-rollout feedback is what drives the observed performance. The missing static-prompt ablation, the construction-induced 92% ceiling, and the large pretraining compute asymmetry all need to be addressed before the sample-complexity advantage can be taken at face value. The paper's strengths include a clearly specified prompting protocol, a concrete success threshold, and a useful qualitative analysis of accumulated in-context examples; these are valuable but they do not by themselves resolve the identified confounds.","major_comments":[{"comment":"The central methodological novelty is the semantic parser 'trained using reinforcement learning and in-context learning,' but the only training mechanism is a verifier that filters candidate expressions by environment success. The paper never runs a static-prompt ablation in which a fixed set of grounded (instruction, expression) examples is provided without any environment feedback or subsequent refinement. Without such a baseline, the observed 92% success rate on the 162 tasks could be fully explained by GPT-4's few-shot inference of the random symbol-to-attribute mapping combined with the pretrained compositional WVFs, making the rollout feedback causally irrelevant to the headline result. This concern is load-bearing for Contribution 2 and for the sample-complexity claim, so the ablation (or a clear demonstration that the feedback loop changes the accepted expression set in a way that improves test performance) is necessary.","section":"§3.2.1, Table 2, Figure 3"},{"comment":"The 92% acceptance threshold is the same number used to report CERLLA's final success rate and to define the oracle upper bound. Since an expression is kept only if its success rate over 100 rollouts is at least 92%, the evaluation curve converging to 92% is partly by construction: the reported success rate is a property of the filter, not an independent confirmation that the parsed expressions match the ground-truth semantics. The paper should report the agreement between accepted expressions and the ground-truth Boolean expressions, or vary the acceptance threshold and show that the final success rate tracks the threshold. Otherwise the statement that CERLLA 'reaches a success rate equal to an oracle policy's upper-bound performance of 92%' is not as informative as it appears.","section":"§3.2.1, Figure 3, Figure 4"},{"comment":"The sample-complexity comparison is confounded by a large pretraining compute asymmetry. CERLLA's WVF pretraining involves nine WVFs, each with 18 CNN-DQN architectures (162 networks in total), plus the Q_min and Q_max networks, trained for 19 million environment steps. The baseline is a single network trained for 21 million steps. Penalizing CERLLA by shifting its curve to start at 19 million steps accounts for the number of environment steps, but not for the per-step compute or the wall-clock cost of training and rolling out 162 value functions. The claim of 'higher success rate and fewer steps' is therefore a claim about environment-step sample complexity under very different architectures, not a practical algorithmic advantage. Please report parameter counts, FLOPs, or wall-clock time, or provide an ablation with a matched-capacity baseline.","section":"§3.1, §4.1, Figure 3"},{"comment":"The held-out generalization experiment assumes that GPT-4, given up to 10 retrieved verified examples, will generate correct Boolean expressions for novel instructions. The paper's own GPT-3.5 results in Table 3 show that the accumulated in-context set can contain 'inconsistent or needlessly complicated' expressions and that this leads to poor generalization. The paper does not quantify how often the BM25 retrieval selects noisy or irrelevant examples for GPT-4, nor how sensitive the results are to the number of in-context examples and the rollout count per candidate expression. Since these are free parameters of the method, a sensitivity analysis (at least on the number of in-context examples and the rollout budget) is needed to establish that the reported 92% is not an artifact of a particular prompt or verification budget.","section":"§4.2, Table 3, Figure 5"}],"minor_comments":[{"comment":"The evaluation protocols for CERLLA and the baseline differ: CERLLA evaluates one randomly selected task every 5,000 environment steps averaged over 100 episodes, while the baseline evaluates all 162 tasks every 50,000 timesteps. This makes the learning curves difficult to compare pointwise, and the statement that CERLLA reaches 92% after 600k steps should be accompanied by a description of the evaluation noise and the exact evaluation schedule used for the baselines.","section":"§4.1, Figure 3"},{"comment":"The sentence 'Each WVF is implemented using |G| = 18 CNN-DQN architectures' is ambiguous about whether the 18 networks are per attribute WVF or shared across all nine attribute WVFs. Since the total number of networks (162) is important for the compute discussion, please state explicitly that the nine attribute WVFs each consist of 18 goal-conditioned networks and clarify where Q_min and Q_max are counted.","section":"§3.1"},{"comment":"The note 'the change in steps scale at 19 million steps' would be clearer with a broken axis or log-scale x-axis; as rendered in the text, it is hard to see the post-pretraining behavior of CERLLA relative to the baseline curve.","section":"Figure 3 caption"},{"comment":"The term 'reinforcement learning' is used for a process that samples candidate expressions, rolls them out, and retains high-success ones; this is closer to verifier-based filtering or evolutionary search than to policy-gradient RL. A brief clarification of why this is called RL would help readers that expect gradient-based updates.","section":"§3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published in TMLR, but as an arXiv submission it still warrants revision under the stated central claims. The missing static-prompt ablation is the most consequential issue because it directly tests the claimed contribution; the compute asymmetry and the self-referential 92% threshold are also serious. None of these issues appear unfixable — adding an ablation, reporting matched-compute numbers, and measuring agreement with ground-truth expressions would substantially strengthen the paper. The GPT-3.5 results in Table 3 are a useful honest signal and should be retained, but they also underscore the need for the sensitivity analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth your time if you care about language-conditioned RL. The core idea—using an LLM to map natural language to Boolean expressions over pretrained world value functions, then verifying candidates by environment rollouts—is genuinely new relative to Shin et al. and Toolformer. The 162-task BabyAI benchmark with held-out generalization is a solid stress test, and the oracle upper bound at 92% is a nice touch that keeps the claims honest. The writing is clear and the related work is handled fairly.\n\nThe soft spots are real, though. First, the parser is not trained with RL in any standard sense. The method samples ten expressions, rolls each out for 100 episodes, and keeps any expression that clears a 92% success bar. That is a verifier, not a reinforcement learning update. The paper's repeated 'RL training' language is misleading and should be corrected.\n\nThe bigger issue is the missing ablation. The stress-test note is on target: we never see a static-prompt baseline where a small set of grounded (instruction, expression) examples is provided up front without any environment feedback. If that static prompt already gets near 92% on the 162 tasks, then the sample-complexity advantage is explained by GPT-4's few-shot parsing plus the compositional value functions, and the environment-rollout loop is causally irrelevant. The paper's central novelty is that loop, so this is load-bearing, not a minor omission. Their own GPT-3.5 results show the accumulated examples can be noisy, which makes the loop's contribution even less clear.\n\nThe compute asymmetry is also worth noting: CERLLA uses 162 DQNs during pretraining versus one network for the baseline. Penalizing by environment steps is fair as far as it goes, but it does not capture the per-step compute gap. The paper would be stronger if it reported compute or at least acknowledged this directly.\n\nNone of this invalidates the main result. The pipeline works, the generalization experiment is convincing, and the comparison to a non-compositional baseline is legitimate. But the contribution of the feedback loop is unquantified, and the paper would be substantially better with that ablation plus corrected terminology. No code or data is released, which also limits reproducibility.\n\nThis paper deserves a serious referee, and I'd bring it to a reading group to discuss the methodology. If I were the reviewer, my main request would be the static-prompt baseline and a clearer statement of what 'RL' means here.","headline":"Worth reading for the compositional value function + LLM parser integration, but the paper overstates the 'RL' learning and never isolates what the environment-rollout feedback actually adds.","tokens_in":13139,"tokens_out":2005,"would_cite":true,"duration_ms":21774,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CERLLA combines compositional value functions with a GPT-4 in-context parser trained by environment rollouts to solve 162 BabyAI language-conditioned tasks at oracle-level 92% success, using far fewer samples than a non-compositional…","keywords":["compositional reinforcement learning","language grounding","semantic parsing","in-context learning","world value functions","Boolean task algebra","sample complexity","compositional generalization"],"falsifier":"Present CERLLA with held-out instructions whose correct Boolean expression requires a combination of three attributes that never co-occurs in any of the ten retrieved verified examples; if success rate falls to the baseline's roughly 80%, the reported generalization is driven by retrieval rather than by learned composition.","tokens_in":12036,"feed_emoji":"🧩","tokens_out":5264,"duration_ms":53913,"temperature":0.7,"pith_summary":"This paper argues that the sample cost of teaching an RL agent many language-conditioned tasks collapses when the policy representation is compositional and the language-to-policy mapping is learned by an LLM whose parses are verified in the environment. The agent, CERLLA, first pretrains a small set of world value functions for basic object attributes, then prompts GPT-4 to translate each instruction into Boolean expressions over those abstract symbols. Candidate expressions are tested by rolling out the composed policy; expressions that reach 92% success become in-context examples for later instructions. On 162 simultaneously learned BabyAI tasks the method matches an oracle's 92% upper bound after roughly 600k post-pretraining steps, while a flat language-conditioned baseline plateaus near 80% after 21 million steps. The paper also reports held-out generalization to instructions never seen during training.","feed_headline":"Compositional agent hits 92% on 162 language-RL tasks","feed_subtitle":"By composing pretrained value functions with an LLM parser trained on rollouts, the method beats a flat baseline's 80%.","key_machinery":"The load-bearing mechanism is the Boolean task algebra over world value functions: for tasks $R$ and $K$, $ar{Q}^*_{R\\vee K}=\\max\\{\\bar{Q}^*_R,\\bar{Q}^*_K\\}$, $ar{Q}^*_{R\\wedge K}=\\min\\{\\bar{Q}^*_R,\\bar{Q}^*_K\\}$, and $ar{Q}^*_{\\neg R}=(\\bar{Q}^*_{\\text{MAX}}+\\bar{Q}^*_{\\text{MIN}})-\\bar{Q}^*_R$. This identity lets any instruction that is a Boolean combination of attribute-level goals be executed by composing nine pretrained value functions with no additional RL. The second mechanism is the in-context parser: GPT-4 proposes ten Boolean expressions, each composed world value function is rolled out for 100 episodes, and any expression with success rate at least 92% is added to the in-context example set, with a length penalty favoring shorter parses.","core_discovery":"The central claim is that a reinforcement learning agent can follow a large set of compositional language instructions by treating language as a Boolean algebra over pretrained world value functions, with the language-to-algebra mapping learned through in-context prompting and environment feedback rather than demonstrations. World value functions extend value functions to penalize reaching unintended goals, so conjunctions, disjunctions, and negations of them yield optimal policies for composed tasks. The semantic parser generates ten candidate Boolean expressions per instruction, instantiates each as a policy, and keeps any expression whose rollout success rate is at least 92%, adding the verified instruction-expression pair to its in-context memory. Because the parser treats the value functions as anonymous symbols, the method is agnostic to what each basis task means and can in principle compose any basis. The paper reports that this pipeline solves all 162 tasks simultaneously, matches the oracle policy's 92% ceiling, and generalizes to held-out instructions, whereas the non-compositional baseline reaches only about 80% success after 21 million steps.","pith_inferences":["The 92% rollout filter is effectively a reward signal for parser training; a learned or cheaper verifier could replace full 100-episode rollouts and allow the method to scale beyond the 162-task BabyAI setting.","Because the parser treats basis symbols as anonymous, the same architecture could compose any set of pretrained skills, such as object-location pairs in robotic pick-and-place, though the paper only tests attribute-based goals.","The paper's GPT-3.5 results indicate that the method's success is tightly coupled to the in-context compositional competence of the LLM; as foundation models improve, the same RL machinery should improve without modification.","The two-phase design, with world value function pretraining separate from parser learning, is a stated limitation; removing it through simultaneous learning would turn CERLLA into a lifelong compositional learner, but that remains untested."],"forward_implications":["A large set of simultaneously learned compositional language-RL tasks can be solved without curricula or demonstrations, provided a compositional value-function basis exists.","With matched environment steps, the compositional method reaches 92% success while the non-compositional baseline reaches about 80%, indicating a substantial sample-complexity advantage.","The agent generalizes to held-out instructions nearly as well as to training instructions, showing that compositional structure in language and in policies transfers together.","The quality of the LLM parser sets the ceiling: with GPT-3.5 the same pipeline accumulates inconsistent or needlessly complicated expressions and fails to beat the baseline, while GPT-4 approaches the oracle bound.","Composed policies are interpretable, since each instruction is reduced to an inspectable Boolean expression over base policies."],"supporting_citations":[{"why":"Supplies the Boolean task algebra and the composition rules for world value functions that the method relies on.","marker":"Nangue Tasse et al. (2020)"},{"why":"Provides the world value function representation and the pretraining recipe that generates the nine basis value functions.","marker":"Nangue Tasse et al. (2022)"},{"why":"Defines the BabyAI environment and the non-compositional baseline architecture against which CERLLA is compared.","marker":"Chevalier-Boisvert et al. (2019)"},{"why":"The GPT-4 model is the large language model used as the in-context semantic parser.","marker":"OpenAI (2023)"},{"why":"The BM25 retrieval algorithm selects up to ten verified in-context examples for each prompt.","marker":"Robertson et al. (2009)"},{"why":"Establishes few-shot LLM semantic parsing, the approach CERLLA extends by adding environment-rollout feedback.","marker":"Shin et al. (2021)"},{"why":"Toolformer demonstrates weak-supervision parser learning with LLMs, a related but distinct alternative to rollout-based supervision.","marker":"Schick et al. (2023)"}],"fun_headline_variants":["RL agent matches oracle ceiling: 92% on 162 composition tasks","Compositional RL beats flat baseline: 92% vs 80% on 162 tasks","One RL agent, 162 language tasks, 92% success, fewer steps","Compositional policy with LLM parser solves 162 tasks at 92%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The main result depends on the assumption that GPT-4, given up to ten retrieved verified examples, will write correct Boolean expressions for instructions it has never seen; the paper's own GPT-3.5 results show this assumption can fail.","fun_headline_variants_meta":{"raw":{"variants":["RL agent matches oracle ceiling: 92% on 162 composition tasks","Compositional RL beats flat baseline: 92% vs 80% on 162 tasks","One RL agent, 162 language tasks, 92% success, fewer steps","Compositional policy with LLM parser solves 162 tasks at 92%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2837,"prompt_tokens":911,"completion_tokens":1926,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1838}},"tokens_in":527,"tokens_out":1926,"duration_ms":15079,"temperature":1.0,"reasoning_tokens":1838,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:05:00.946374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present CERLLA with held-out instructions whose correct Boolean expression requires a combination of three attributes that never co-occurs in any of the ten retrieved verified examples; if success rate falls to the baseline's roughly 80%, the reported generalization is driven by retrieval rather than by learned composition.","supporting_citations":[{"cited_title":"Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil An- tonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Ben Van Durme","cited_arxiv_id":null,"evidence_quote":"Establishes few-shot LLM semantic parsing, the approach CERLLA extends by adding environment-rollout feedback."}],"review_version":1}