Pith. sign in

REVIEW 4 major objections 4 minor 14 references

Compositional Instruction Following with Language Models and Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CERLLA combines compositional value functions with a GPT-4 in-context parser trained by environment rollouts to solve 162 BabyAI language-conditioned tasks at oracle-level 92% success, using far fewer samples than a non-compositional…

desk verdict Worth reading for the compositional value function + LLM parser integration, but the paper overstates the 'RL' learning and never isolates what the environment-rollout feedback actually adds. read the letter →

arxiv 2501.12539 v1 pith:6H3H7CQA submitted 2025-01-21 cs.LG cs.CL

Vanya Cohen , Geraud Nangue Tasse , Nakul Gopalan , Steven James , Matthew Gombolay , Ray Mooney , Benjamin Rosman This is my paper · ORCID
classification cs.LGcs.CL
keywords compositionalreinforcementlearninglanguagegroundingsemanticparsingin-contextworldvaluefunctionsBooleantaskalgebrasamplecomplexitygeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the sample cost of teaching an RL agent many language-conditioned tasks collapses when the policy representation is compositional and the language-to-policy mapping is learned by an LLM whose parses are verified in the environment. The agent, CERLLA, first pretrains a small set of world value functions for basic object attributes, then prompts GPT-4 to translate each instruction into Boolean expressions over those abstract symbols. Candidate expressions are tested by rolling out the composed policy; expressions that reach 92% success become in-context examples for later instructions. On 162 simultaneously learned BabyAI tasks the method matches an oracle's 92% upper bound after roughly 600k post-pretraining steps, while a flat language-conditioned baseline plateaus near 80% after 21 million steps. The paper also reports held-out generalization to instructions never seen during training.

What carries the argument

The load-bearing mechanism is the Boolean task algebra over world value functions: for tasks $R$ and $K$, $ar{Q}^*_{R\vee K}=\max\{\bar{Q}^*_R,\bar{Q}^*_K\}$, $ar{Q}^*_{R\wedge K}=\min\{\bar{Q}^*_R,\bar{Q}^*_K\}$, and $ar{Q}^*_{\neg R}=(\bar{Q}^*_{\text{MAX}}+\bar{Q}^*_{\text{MIN}})-\bar{Q}^*_R$. This identity lets any instruction that is a Boolean combination of attribute-level goals be executed by composing nine pretrained value functions with no additional RL. The second mechanism is the in-context parser: GPT-4 proposes ten Boolean expressions, each composed world value function is rolled out for 100 episodes, and any expression with success rate at least 92% is added to the in-context example set, with a length penalty favoring shorter parses.

What would settle it

Present CERLLA with held-out instructions whose correct Boolean expression requires a combination of three attributes that never co-occurs in any of the ten retrieved verified examples; if success rate falls to the baseline's roughly 80%, the reported generalization is driven by retrieval rather than by learned composition.

Watch

Extended reading notes

Core claim

The central claim is that a reinforcement learning agent can follow a large set of compositional language instructions by treating language as a Boolean algebra over pretrained world value functions, with the language-to-algebra mapping learned through in-context prompting and environment feedback rather than demonstrations. World value functions extend value functions to penalize reaching unintended goals, so conjunctions, disjunctions, and negations of them yield optimal policies for composed tasks. The semantic parser generates ten candidate Boolean expressions per instruction, instantiates each as a policy, and keeps any expression whose rollout success rate is at least 92%, adding the verified instruction-expression pair to its in-context memory. Because the parser treats the value functions as anonymous symbols, the method is agnostic to what each basis task means and can in principle compose any basis. The paper reports that this pipeline solves all 162 tasks simultaneously, matches the oracle policy's 92% ceiling, and generalizes to held-out instructions, whereas the non-compositional baseline reaches only about 80% success after 21 million steps.

Load-bearing premise

The main result depends on the assumption that GPT-4, given up to ten retrieved verified examples, will write correct Boolean expressions for instructions it has never seen; the paper's own GPT-3.5 results show this assumption can fail.

Editorial extensions

If this is right

  • A large set of simultaneously learned compositional language-RL tasks can be solved without curricula or demonstrations, provided a compositional value-function basis exists.
  • With matched environment steps, the compositional method reaches 92% success while the non-compositional baseline reaches about 80%, indicating a substantial sample-complexity advantage.
  • The agent generalizes to held-out instructions nearly as well as to training instructions, showing that compositional structure in language and in policies transfers together.
  • The quality of the LLM parser sets the ceiling: with GPT-3.5 the same pipeline accumulates inconsistent or needlessly complicated expressions and fails to beat the baseline, while GPT-4 approaches the oracle bound.
  • Composed policies are interpretable, since each instruction is reduced to an inspectable Boolean expression over base policies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 92% rollout filter is effectively a reward signal for parser training; a learned or cheaper verifier could replace full 100-episode rollouts and allow the method to scale beyond the 162-task BabyAI setting.
  • Because the parser treats basis symbols as anonymous, the same architecture could compose any set of pretrained skills, such as object-location pairs in robotic pick-and-place, though the paper only tests attribute-based goals.
  • The paper's GPT-3.5 results indicate that the method's success is tightly coupled to the in-context compositional competence of the LLM; as foundation models improve, the same RL machinery should improve without modification.
  • The two-phase design, with world value function pretraining separate from parser learning, is a stated limitation; removing it through simultaneous learning would turn CERLLA into a lifelong compositional learner, but that remains untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript introduces CERLLA, a method that combines pretrained compositional world value functions (WVFs) with an LLM-based semantic parser for language-conditioned RL tasks. In a first phase, nine attribute-level WVFs are pretrained, each implemented with 18 CNN-DQN heads. In a second phase, GPT-4 (or GPT-3.5) is prompted with up to 10 BM25-retrieved in-context examples to produce 10 candidate Boolean expressions for a natural-language mission; each candidate is instantiated as a composed WVF and rolled out for 100 episodes, and expressions with success rate at least 92% are added to the in-context example set. The paper claims that CERLLA solves 162 simultaneous BabyAI tasks with a 92% success rate, matching an oracle upper bound, while a non-compositional baseline reaches only about 80% after 21 million steps, and that CERLLA generalizes to held-out tasks. The central assertion is that the environment-rollout feedback loop, together with compositional value functions, yields a sample-complexity advantage over learning a single joint policy.

Significance. If the central claims hold, the paper makes a useful contribution by connecting compositional value-function algebra with in-context LLM parsing, and it reports a large-scale (162-task) empirical study with an interpretable decomposition. The paper is honest about the GPT-3.5 failure mode and includes a limitations section. However, the evidence as presented does not yet establish the load-bearing causal claim that environment-rollout feedback is what drives the observed performance. The missing static-prompt ablation, the construction-induced 92% ceiling, and the large pretraining compute asymmetry all need to be addressed before the sample-complexity advantage can be taken at face value. The paper's strengths include a clearly specified prompting protocol, a concrete success threshold, and a useful qualitative analysis of accumulated in-context examples; these are valuable but they do not by themselves resolve the identified confounds.

major comments (4)
  1. [§3.2.1, Table 2, Figure 3] The central methodological novelty is the semantic parser 'trained using reinforcement learning and in-context learning,' but the only training mechanism is a verifier that filters candidate expressions by environment success. The paper never runs a static-prompt ablation in which a fixed set of grounded (instruction, expression) examples is provided without any environment feedback or subsequent refinement. Without such a baseline, the observed 92% success rate on the 162 tasks could be fully explained by GPT-4's few-shot inference of the random symbol-to-attribute mapping combined with the pretrained compositional WVFs, making the rollout feedback causally irrelevant to the headline result. This concern is load-bearing for Contribution 2 and for the sample-complexity claim, so the ablation (or a clear demonstration that the feedback loop changes the accepted expression set in a way that improves test performance) is necessary.
  2. [§3.2.1, Figure 3, Figure 4] The 92% acceptance threshold is the same number used to report CERLLA's final success rate and to define the oracle upper bound. Since an expression is kept only if its success rate over 100 rollouts is at least 92%, the evaluation curve converging to 92% is partly by construction: the reported success rate is a property of the filter, not an independent confirmation that the parsed expressions match the ground-truth semantics. The paper should report the agreement between accepted expressions and the ground-truth Boolean expressions, or vary the acceptance threshold and show that the final success rate tracks the threshold. Otherwise the statement that CERLLA 'reaches a success rate equal to an oracle policy's upper-bound performance of 92%' is not as informative as it appears.
  3. [§3.1, §4.1, Figure 3] The sample-complexity comparison is confounded by a large pretraining compute asymmetry. CERLLA's WVF pretraining involves nine WVFs, each with 18 CNN-DQN architectures (162 networks in total), plus the Q_min and Q_max networks, trained for 19 million environment steps. The baseline is a single network trained for 21 million steps. Penalizing CERLLA by shifting its curve to start at 19 million steps accounts for the number of environment steps, but not for the per-step compute or the wall-clock cost of training and rolling out 162 value functions. The claim of 'higher success rate and fewer steps' is therefore a claim about environment-step sample complexity under very different architectures, not a practical algorithmic advantage. Please report parameter counts, FLOPs, or wall-clock time, or provide an ablation with a matched-capacity baseline.
  4. [§4.2, Table 3, Figure 5] The held-out generalization experiment assumes that GPT-4, given up to 10 retrieved verified examples, will generate correct Boolean expressions for novel instructions. The paper's own GPT-3.5 results in Table 3 show that the accumulated in-context set can contain 'inconsistent or needlessly complicated' expressions and that this leads to poor generalization. The paper does not quantify how often the BM25 retrieval selects noisy or irrelevant examples for GPT-4, nor how sensitive the results are to the number of in-context examples and the rollout count per candidate expression. Since these are free parameters of the method, a sensitivity analysis (at least on the number of in-context examples and the rollout budget) is needed to establish that the reported 92% is not an artifact of a particular prompt or verification budget.
minor comments (4)
  1. [§4.1, Figure 3] The evaluation protocols for CERLLA and the baseline differ: CERLLA evaluates one randomly selected task every 5,000 environment steps averaged over 100 episodes, while the baseline evaluates all 162 tasks every 50,000 timesteps. This makes the learning curves difficult to compare pointwise, and the statement that CERLLA reaches 92% after 600k steps should be accompanied by a description of the evaluation noise and the exact evaluation schedule used for the baselines.
  2. [§3.1] The sentence 'Each WVF is implemented using |G| = 18 CNN-DQN architectures' is ambiguous about whether the 18 networks are per attribute WVF or shared across all nine attribute WVFs. Since the total number of networks (162) is important for the compute discussion, please state explicitly that the nine attribute WVFs each consist of 18 goal-conditioned networks and clarify where Q_min and Q_max are counted.
  3. [Figure 3 caption] The note 'the change in steps scale at 19 million steps' would be clearer with a broken axis or log-scale x-axis; as rendered in the text, it is hard to see the post-pretraining behavior of CERLLA relative to the baseline curve.
  4. [§3.2.1] The term 'reinforcement learning' is used for a process that samples candidate expressions, rolls them out, and retains high-success ones; this is closer to verifier-based filtering or evolutionary search than to policy-gradient RL. A brief clarification of why this is called RL would help readers that expect gradient-based updates.

Circularity Check

1 steps flagged · score 2.0 of 10

Mostly self-contained; one self-referential 92% acceptance threshold makes the headline 'matches oracle' true by construction.

  1. self definitional [Section 3.2.1 (In-Context Semantic Parsing with RL); Figure 1 caption; Section 5 Discussion]
    "If the success rate across these episodes in reaching the specified goals is greater than or equal to92%, the language instruction and Boolean expression are added to the list of in-context examples."

    The acceptance rule defines a 'valid parse' as one whose rollout success rate is at least 92%, and the evaluation metric is the same rollout success rate. Therefore, for every task whose expression is accepted, the agent's later measured success rate is >=92% by construction. The paper's headline that CERLLA 'attains a 92% success rate (matching the performance upper bound of the Oracle Agent)' is thus not an independent empirical discovery: the 92% figure is imposed by the threshold used to accept parses. The substantive empirical content is that the agent finds accepted expressions for most of the 162 tasks and does so with fewer environment steps than the baseline; the exact equality to the oracle ceiling is a calibration artifact, not a measured outcome.

full rationale

The paper's central claim is that environment-rollout feedback can train an LLM to map instructions to Boolean compositions of pretrained world value functions, solving 162 tasks with lower sample complexity than a non-compositional baseline. This is not circular: the rollout success rate is an external environment signal, the baseline comparison is empirical, and the held-out generalization experiment tests genuinely novel instructions. The compositional WVF algebra is cited from prior work by the same authors, but it is a mathematical foundation with explicit definitions and stated assumptions, not a fitted version of the present result, and no uniqueness theorem is invoked to forbid alternative approaches. The one genuinely self-referential element is the 92% acceptance threshold: expressions are retained only if their rollout success rate is >=92%, so the reported final success rate of 92% (equated to the oracle ceiling) is guaranteed by the acceptance rule for any solved task. This is a calibration choice rather than an independent discovery. The central sample-complexity advantage does not reduce to this threshold, because the baseline is not subject to the same filter and still fails to reach 92%; nevertheless, the specific 'matches oracle' figure is true by construction. Overall circularity is low, so the score is 2 rather than 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the prior Boolean task algebra for value-function composition, on the semantic competence of GPT-4, and on the reliability of the 92% success-rate filter. The free parameters are hand-set hyperparameters rather than fitted scientific constants. No new physical or conceptual entities are introduced beyond abstract symbols for the pretrained skills.

free parameters (3)
  • 92% success threshold = 0.92
    Hand-set as the acceptance criterion for a valid parse and as the reported target; it matches the empirically measured oracle ceiling.
  • Number of in-context examples = 10
    The prompt includes up to 10 retrieved examples, chosen by hand.
  • Rollout count per candidate expression = 100 episodes
    Number of episodes used to estimate the success rate of each candidate expression, chosen to balance cost and reliability.
assumptions (4)
  • domain assumption Boolean composition of value functions (min, max, negation) yields optimal policies for composed tasks.
    Inherited from Nangue Tasse et al. (2020); the paper builds on this without re-deriving it (Section 2).
  • standard math The extended reward function in Eq. 1 correctly penalizes reaching unintended goal states, making the world value function formulation valid.
    Standard construction for world value functions, presented in Section 2.
  • domain assumption GPT-4 has sufficient pretrained semantic competence to map novel instructions to Boolean expressions given a few in-context examples.
    This is the load-bearing assumption for generalization; the GPT-3.5 results show it can fail (Section 5, Figure 5).
  • ad hoc to paper A 92% success rate over 100 rollouts is a reliable indicator that a Boolean expression correctly captures the instruction.
    Used to filter parses; no probabilistic justification or error analysis is provided (Section 3.2.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Compositional Instruction Following with Language Models and Reinforcement Learning." pith.science (2026). https://pith.science/paper/6H3H7CQA

@misc{pith2026250112539,
  author       = {Pith},
  title        = {Pith review of: Compositional Instruction Following with Language Models and Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6H3H7CQA}},
  note         = {Machine review of arXiv:2501.12539}
}
read the original abstract

Combining reinforcement learning with language grounding is challenging as the agent needs to explore the environment while simultaneously learning multiple language-conditioned tasks. To address this, we introduce a novel method: the compositionally-enabled reinforcement learning language agent (CERLLA). Our method reduces the sample complexity of tasks specified with language by leveraging compositional policy representations and a semantic parser trained using reinforcement learning and in-context learning. We evaluate our approach in an environment requiring function approximation and demonstrate compositional generalization to novel tasks. Our method significantly outperforms the previous best non-compositional baseline in terms of sample complexity on 162 tasks designed to test compositional generalization. Our model attains a higher success rate and learns in fewer steps than the non-compositional baseline. It reaches a success rate equal to an oracle policy's upper-bound performance of 92%. With the same number of environment steps, the baseline only reaches a success rate of 80%.

Figures

Figures reproduced from arXiv: 2501.12539 by the authors.

Figure 1
Figure 1. Pipeline diagram of the learning process for the CERLLA agent. The agent takes in a BabyAI [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example of a task in the BabyAI domain (Chevalier-Boisvert et al., 2019). The agent (red triangle) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Results for learning all 162 tasks simultaneously. The mean episode success rate is plotted against the number of environment steps. Learning curves are presented for CERLLA and the non-compositional baseline agents. The Oracle agent is given the ground-truth Boolean expressions and upper bounds the attainable success rate in the environment, denoted by the dashed line at 92%. Our method is initialized at 19 million… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The mean number of tasks solved is plotted against the number of environment steps. This [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Generalization performance for learning on [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages

  1. [5]

    URL https://doi.org/10.18653/v1/n19-1423

    doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423. Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378,

    arXiv

  2. [6]

    What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal manage- ment and action execution

    Juraj Dzifcak, Matthias Scheutz, Chitta Baral, and Paul Schermerhorn. What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal manage- ment and action execution. In 2009 IEEE International Conference on Robotics and Automation, pp. 4163–4168. IEEE,

    work page 2009

  3. [7]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In3rd International Conference on Learning Representations, ICLR 2015,

    work page 2015

  4. [8]

    How to reuse and compose knowledge for a lifetime of tasks: A survey on continual learning and functional composition

    12 Published in Transactions on Machine Learning Research (12/2024) Jorge Mendez-Mendez and Eric Eaton. How to reuse and compose knowledge for a lifetime of tasks: A survey on continual learning and functional composition. Transactions on Machine Learning Research (TMLR),

    work page 2024

  5. [9]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11

    2019

  6. [10]

    Stephen Robertson, Hugo Zaragoza, et al

    URLhttps://arxiv.org/abs/1908.10084. Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389,

    arXiv 1908

  7. [11]

    Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil An- tonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Ben Van Durme

    Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil An- tonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Ben Van Durme. Constrained language models yield few-shot semantic parsers. In 2021 Empirical Methods in Natural Lan- guage Processing, November

    work page 2021

  8. [14]

    As shown, many of these expressions are not consistent or needlessly complicated

    A Appendix Language Instruction Boolean Expression pick up a ball or Symbol_0| Symbol_4 a grey object pick up a box that ∼Symbol_4 & Symbol_2 is not grey pick up a grey ball Symbol_0 & Symbol_4 pick up a ball or an (Symbol_0| Symbol_1)| object that is not grey ∼Symbol_4 pick up a ball that Symbol_0 & ∼Symbol_4 is not grey pick up a grey object or (Symbol_...

    work page 2015

Show all 14 references
  1. [2018]

    Transporter networks: Rearranging the visual world for robotic manipulation.Conference on Robot Learning (CoRL),

    13 Published in Transactions on Machine Learning Research (12/2024) Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Arm- strong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter networks: Rearranging the vis...

    2024

  2. [2019]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.CoRR, abs/2306.13831,

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.CoRR, abs/2306.13831,

    arXiv

  3. [2020]

    Rt-1: Robotics transformer for real- world control at scale.arXiv preprint arXiv:2212.06817,

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real- world control at scale.arXiv preprint arXiv:2212.06817,

    arXiv

  4. [2021]

    Mpnet: Masked and permuted pre-training for language understanding.arXiv preprint arXiv:2004.09297,

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding.arXiv preprint arXiv:2004.09297,

    2004 arXiv

  5. [2022]

    Eager: Asking and answering questions for automatic reward shaping in language-guided rl.Advances in Neural Information Processing Systems, 35:12478–12490,

    11 Published in Transactions on Machine Learning Research (12/2024) Thomas Carta, Pierre-Yves Oudeyer, Olivier Sigaud, and Sylvain Lamprier. Eager: Asking and answering questions for automatic reward shaping in language-guided rl.Advances in Neural Information Processing Syste...

    2024

  6. [2023]

    BERT: pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of...

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.