Pith. sign in

REVIEW 4 major objections 4 minor 19 references

LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LeDeepChef, a deep reinforcement learning agent that ranks a pruned set of high-level commands, earns 69.3 percent of achievable points on unseen cooking-game test games and beats standard text-game baselines.

desk verdict A genuinely new action-space reduction for text-based games with a solid external benchmark result, but the unverified coverage of its hand-crafted command generator keeps the generalization claim from being fully established. read the letter →

arxiv 1909.01646 v1 pith:4T5ZGHK6 submitted 2019-09-04 cs.LG cs.AIstat.ML

Leonard Adolphs , Thomas Hofmann This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords text-basedgamesdeepreinforcementlearningaction-spacepruninghierarchicalgeneralizationTextWorldactor-criticrecipemanager
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a deep reinforcement learning agent can generalize across a whole family of text-based games, not just solve one game, if the action space is aggressively pruned into high-level commands. The agent, LeDeepChef, uses an actor-critic policy that ranks a short list of candidate commands each turn, where the list is produced by a hand-designed recipe module, a navigation module, and fixed rules. On unseen games from the First TextWorld Problems challenge it earns 69.3 percent of achievable points, while standard text-game baselines stay below 15 percent. The broader claim is that for structured natural-language tasks, shrinking the action space may matter more than the choice of reinforcement learning algorithm.

What carries the argument

The load-bearing mechanism is command generation: a fixed pipeline that collapses a combinatorial natural-language action space into a few candidates per turn. The Recipe Manager is a supervised GRU classifier trained on recipes augmented with a broad food-item database; given the recipe and inventory text it labels each direction as still-needed or done, then emits high-level commands such as take all required ingredients from here and drop unnecessary items, plus specific cooking commands only when the ingredient and required appliance are present. The Navigator is a supervised GRU that reads a room description, predicts which cardinal directions connect to rooms, and finds multi-word closed-door names, adding go and open commands accordingly. Fixed rules add look, inventory, examine cookbook, prepare meal, and eat meal under simple conditions. The paper identifies this pruning as the most important step: it turns an intractable exploration problem into a stable ranking problem for the actor-critic network.

What would settle it

Generate a large batch of new games in the same family and compute an oracle solution for each; if any oracle solution contains a command outside the set produced by the recipe manager, navigator, and fixed rules, then the central generalization claim fails for that game. More directly, run the agent with a perfect policy that always picks the optimal available command: the resulting score is an upper bound set by command coverage, and comparing that bound with the current 69.3 percent measures how much headroom the pruning leaves.

Watch

Extended reading notes

Core claim

The central claim is that a family-level text-game agent can be built by treating every turn as a ranking problem over a small, context-dependent command set, rather than as an open-ended language-generation problem. LeDeepChef encodes eight pieces of textual context with bidirectional GRUs, keeps a temporal hidden state, and scores each candidate command with an MLP. The candidates come from a command-generation layer that combines a supervised Recipe Manager, a supervised Navigator, and fixed rules. Trained with an advantage actor-critic objective, the agent reaches 74.4 percent of achievable points on the unseen validation games and 69.3 percent on the unseen test set, while LSTM-DQN and DRRN baselines remain below 15 percent and the closest TextWorld-specific comparison reports 58 percent on a different validation split.

Load-bearing premise

The command generator's fixed rules and trained modules must, in every game, include at least one action that lies on a successful path; if an unseen game needs an action the pruned lists never offer, the agent cannot solve it no matter how well the policy learns.

Editorial extensions

If this is right

  • If LeDeepChef's design is right, then for any natural-language task with a structured goal the bottleneck is the command interface: shrinking the action set to a few well-chosen high-level options can turn an unsolvable exploration problem into a learnable ranking problem.
  • The random baseline on the same pruned commands earns about 39.6 percent on the test set, so command generation alone provides a strong floor and leaves room for the learned policy to add roughly 30 more points.
  • Because the recipe model is trained on recipes augmented with a large food-item database, the agent should keep working when new recipes name ingredients it never saw during training, as long as those ingredients lie close to food words in the embedding space.
  • Actor-critic training appears to contribute beyond pruning: the DRRN baseline on the same pruned commands plateaued near 44 percent on the test set, whereas LeDeepChef reached 69.3 percent.
  • The approach suggests that standardized text-game benchmarks should report performance relative to achievable points on unseen game families, since single-game accuracy does not measure generalization across rooms, objects, and task descriptions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is to measure oracle coverage of the command generator: solve each game with a planner and check whether every optimal trajectory's actions are present in the pruned set; this would separate the contribution of command coverage from the policy's ranking ability.
  • The same action-space pruning idea could transfer to task-oriented dialogue, where exposing a small set of intent and slot commands may outperform free-form text generation because exploration is concentrated on a tractable set.
  • Because the modules are partly hand-designed, an ablation that removes the recipe manager, the navigator, or the high-level grouping would reveal how much of the gain comes from domain knowledge versus learning, which the paper does not report.
  • A learned replacement for the fixed command rules would show whether the approach scales to game families whose required actions cannot be enumerated by a human designer ahead of time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents LeDeepChef, a deep reinforcement learning agent for families of text-based games, specifically the cooking-themed games in Microsoft's First TextWorld Problems challenge. The agent combines an advantage actor-critic architecture with a hand-designed command generation pipeline that prunes the action space via a supervised Recipe Manager, a supervised Navigator, and fixed rules. The agent is evaluated on the official challenge validation and test sets, achieving 74.4% and 69.3% of achievable points respectively, and placing second overall. The authors also compare against standard TBG baselines (LSTM-DQN, DRRN) and a pruned-action variant of DRRN, reporting substantially higher scores.

Significance. If the reported results are taken at face value, the paper demonstrates that combining RL with hand-crafted action-space abstraction and supervised helper modules can generalize across a family of unseen text-based games, and the evaluation on the official external TextWorld test set gives the central empirical claim strong external validity. The released code is also a positive contribution. However, the significance of the architectural contribution is currently limited because the main design choices—high-level command grouping, the learned command-generation modules, and the policy architecture—are not isolated by ablations, and because the correctness of the pruning pipeline is never audited.

major comments (4)
  1. [Command Generation; Results (Table 2)] The central result depends on the unverified premise that the command-generation pipeline (Recipe Manager, Navigator, and the fixed rules in Table 1) always places at least one optimal action in the candidate set at every state. The Recipe Manager is a binary classifier over recipe directions, and the Navigator must identify every closed door; a single false negative silently removes the only action that can progress the game, capping the achievable score regardless of policy quality. The paper reports no accuracy, false-negative rate, or coverage analysis for either learned module, and no per-game audit of whether the optimal trajectory was contained in the candidate set. This is load-bearing for the claim of generalization: the 69.3% test score is an upper bound on the product of coverage probability and policy performance given coverage. Please add error analyses for the Recipe Manager and Navigator, and a coverage audit on the validation/test games (for example, by checking whether the candidate set contains the gold actions on held-out walkthroughs).
  2. [Results (Table 2, Figure 3)] The baseline comparison is weakened by the fact that LSTM-DQN and DRRN were trained for only 3 epochs. The text states that these baselines 'do not exceed the 20% mark ... during 3 epochs of training' and concludes they are not suitable for the family-of-games task. This conflates a limited training budget with an inability to learn. Since the paper's headline claim is that the agent outperforms standard TBG baselines, the baselines should be trained to convergence or at least for a comparable number of updates, with the training budget reported. As written, the comparison does not support the strong conclusion drawn from it.
  3. [Results (Table 2)] The comparison to Yin and May (2019b) is not controlled: that model's 58% is reported on the authors' own validation set (hold-out data from the official training set), which the paper describes as 'comparable in difficulty' but which is not the official validation or test set used for LeDeepChef and the other baselines. Describing this as evidence that LeDeepChef 'generalizes best' is therefore not justified by the table. This comparison should be removed, or the Yin and May model should be evaluated on the same validation/test games.
  4. [Results (Comparison on pruned commands)] The comparison between LeDeepChef and DRRN Pruned conflates at least three differences: the grouped high-level commands, the richer context feature set, and the actor-critic objective versus DQN. The paper attributes the gap to all three reasons (i)-(iii) but provides no ablation to isolate the contribution of the high-level command grouping, which is one of the stated contributions. Without an ablation that keeps the action set and architecture fixed while varying only the high-level grouping, the claim that abstracting to high-level commands improves generalization is not demonstrated.
minor comments (4)
  1. [Agent (Training)] Equation (1) uses the notation R_t with both a discount exponent T - t and a sum to T - t, which is non-standard and slightly confusing; please clarify that the sum includes the immediate reward and the bootstrapped value at T, and use consistent indices.
  2. [Table 1] Table 1 has minor wording issues: 'eat meal Added if meal is in agent\'s inventory' and 'prepare meal Added once ...' should read 'Added when ...' for readability.
  3. [Command Generation (Recipe Commands)] The description of the self-constructed training dataset for the Recipe Manager does not report its size or the number of augmented ingredients; please include these details for reproducibility, along with the exact data-generation procedure.
  4. [Results (Table 2)] The standard deviations in Table 2 are reported over ten seeds, but the baseline rows for LSTM-DQN show ±.00 on both metrics; this is visually indistinguishable from zero and should be reported with more significant figures.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: main result is benchmarked against external hidden games and does not reduce to its inputs.

full rationale

The paper's central claim is that LeDeepChef generalizes to unseen TextWorld games and scores 69.3% of achievable points on the challenge test set. The evaluation is performed by the external TextWorld game engine on hidden games, and the reported score is not defined in terms of the agent's own fitted components. The command-generation pipeline (Recipe Manager, Navigator, and the fixed rules in Table 1) is a hand-designed and supervised-learned action-space pruning mechanism, but it is not fit to the test-set scores and no equation or construction makes the final score equal to the output of the recipe classifier or navigator. The paper compares against baselines on the same pruned command set, which isolates the contribution of the actor-critic ranking and high-level commands. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known result as a derivation. The unverified coverage assumption about the pruned action space is a legitimate robustness or correctness concern, not circularity: even if command generation occasionally omits an optimal action, the agent's score is still an externally measured outcome of a system whose components are not defined in terms of that outcome. The derivation chain is therefore self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on hand-chosen architecture dimensions, an unreleased training data distribution, and the completeness of a hand-crafted action space. The supervised recipe and navigation modules are integral to the method but their individual accuracies are not reported.

free parameters (6)
  • context GRU hidden size = 32
    Chosen by hand; each of the eight context features is encoded to 32 dimensions, concatenated to 256.
  • MLP hidden size = 256
    Chosen by hand for both value and scoring MLPs; no hyperparameter search reported for the main model.
  • word embedding dimension = 100
    Uses 100-dimensional pre-trained GloVe embeddings; trainable thereafter.
  • discount factor gamma = not reported
    Equation (1) defines the n-step TD return with gamma, but the value used for LeDeepChef is not given; only DRRN baseline uses 0.9.
  • recipe model classification threshold = 0.5 implicit
    The recipe manager outputs probabilities for each direction; the decision threshold is not stated.
  • training budget = 3 epochs (about 12600 episodes)
    The agent and baselines are trained for 3 epochs; this budget may disadvantage baselines with slower convergence.
assumptions (5)
  • domain assumption The TextWorld game engine accepts and correctly executes all low-level commands generated by the recipe manager and navigator (e.g., 'take red hot pepper', 'cook carrot with oven').
    The command generation assumes the constructed commands are syntactically valid and semantically correct in the game; no parser validation is reported.
  • domain assumption The self-constructed recipe/inventory dataset, augmented with freebase food items, is representative of the challenge's test distribution.
    The recipe manager is trained on this data; if test recipes contain unseen phrasings or ingredients outside the augmentation, the classifier may mislabel directions.
  • domain assumption Pre-trained GloVe embeddings provide enough semantic proximity for unseen ingredients to generalize.
    The paper relies on GloVe to handle unseen food items, but no analysis of coverage or nearest-neighbor behavior is given.
  • ad hoc to paper The hand-crafted high-level commands and fixed rules (Table 1) cover all actions required to solve every game in the family.
    The policy can only select from the generated command set; any necessary action outside this set makes game completion impossible.
  • domain assumption The environment's reward is fully captured by the score increments and the 100-step limit, making the n-step TD return a valid training signal.
    The actor-critic objective in Eq. (1) assumes a Markovian reward process with bounded horizon; this holds for the TextWorld challenge games.
invented entities (1)
  • High-level command abstraction (e.g., 'take all required ingredients from here', 'drop unnecessary items')
    purpose: Reduce the action space by grouping multiple low-level actions into a single abstract choice, which the recipe manager expands into concrete parser commands.
    No standalone falsifiable prediction; the only evidence is the in-paper comparison to DRRN Pruned, which is confounded by architecture differences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games." pith.science (2026). https://pith.science/paper/4T5ZGHK6

@misc{pith2026190901646,
  author       = {Pith},
  title        = {Pith review of: LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4T5ZGHK6}},
  note         = {Machine review of arXiv:1909.01646}
}
read the original abstract

While Reinforcement Learning (RL) approaches lead to significant achievements in a variety of areas in recent history, natural language tasks remained mostly unaffected, due to the compositional and combinatorial nature that makes them notoriously hard to optimize. With the emerging field of Text-Based Games (TBGs), researchers try to bridge this gap. Inspired by the success of RL algorithms on Atari games, the idea is to develop new methods in a restricted game world and then gradually move to more complex environments. Previous work in the area of TBGs has mainly focused on solving individual games. We, however, consider the task of designing an agent that not just succeeds in a single game, but performs well across a whole family of games, sharing the same theme. In this work, we present our deep RL agent--LeDeepChef--that shows generalization capabilities to never-before-seen games of the same family with different environments and task descriptions. The agent participated in Microsoft Research's "First TextWorld Problems: A Language and Reinforcement Learning Challenge" and outperformed all but one competitor on the final test set. The games from the challenge all share the same theme, namely cooking in a modern house environment, but differ significantly in the arrangement of the rooms, the presented objects, and the specific goal (recipe to cook). To build an agent that achieves high scores across a whole family of games, we use an actor-critic framework and prune the action-space by using ideas from hierarchical reinforcement learning and a specialized module trained on a recipe database.

Figures

Figures reproduced from arXiv: 1909.01646 by the authors.

Figure 1
Figure 1. Simple game that shows the basic structure of the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the model. From a textual description of the context, together with [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of our model to several baseline models on the TextWorld challenge games, as points per game relative [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Illustration of the recipe manager [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

  1. [2]

    J.; Asri, L

    C ˆot´e, M.; K´ad´ar, ´A.; Yuan, X.; Kybartas, B.; Barnes, T.; Fine, E.; Moore, J.; Hausknecht, M. J.; Asri, L. E.; Adada, M.; Tay, W.; and Trischler, A. 2018. Textworld: A learning environment for text-based games. CoRR abs/1806.11532

    arXiv 2018

  2. [5]

    What can you do with a rock? Affordance extraction via word embeddings

    What can you do with a rock? affordance extraction via word embeddings. CoRR abs/1703.03429

    work page Pith review arXiv

  3. [9]

    Kostka, B.; Kwiecien, J.; Kowalski, J.; and Rych- likowski, P. 2017. Text-based adventures of the golovin AI agent. CoRR abs/1705.05637

    work page Pith review arXiv 2017

  4. [12]

    CoRR abs/1602.01783

    Asynchronous methods for deep reinforcement learn- ing. CoRR abs/1602.01783

    arXiv

  5. [13]

    D.; and Barzilay, R

    Narasimhan, K.; Kulkarni, T. D.; and Barzilay, R

    work page

  6. [14]

    CoRR abs/1506.08941

    Language understanding for text-based games using deep reinforcement learning. CoRR abs/1506.08941

    arXiv

  7. [16]

    S., and Barto, A

    Sutton, R. S., and Barto, A. G. 2018. Reinforcement Learning: An Introduction. The MIT Press, second edition

    work page 2018

  8. [17]

    Towards Solving Text-based Games by Producing Adaptive Action Spaces

    Tao, R. Y .; Cˆot´e, M.; Yuan, X.; and Asri, L. E. 2018. Towards solving text-based games by producing adaptive ac- tion spaces. CoRR abs/1812.00855

    work page Pith review arXiv 2018

Show all 19 references
  1. [18]

    Vinyals, O.; Fortunato, M.; and Jaitly, N. 2015. Pointer networks. In Cortes, C.; Lawrence, N. D.; Lee, D. D.; Sugiyama, M.; and Garnett, R., eds., Advances in Neural Information Processing Systems 28 . Curran Asso- ciates, Inc. 2692–2700. [2019a] Yin, X., and May, J. 2019a. C...

    2015 arXiv

  2. [19]

    Ys84WnppVT4GhMM21wnx/hn0Xxw=

    Zahavy, T.; Haroush, M.; Merlis, N.; Mankowitz, D. J.; and Mannor, S. 2018. Learn what not to learn: Action elimination with deep reinforcement learning. In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., Advances in Neural Inform...

    2018

  3. [1980]

    Infocom. 1980. Zork i

    1980

  4. [1993]

    Dayan, P., and Hinton, G. E. 1993. Feudal reinforce- ment learning. In Hanson, S. J.; Cowan, J. D.; and Giles, C. L., eds., Advances in Neural Information Processing Sys- tems 5. Morgan-Kaufmann. 271–278

    1993

  5. [1997]

    Hochreiter, S., and Schmidhuber, J. 1997. Long short- term memory. Neural Comput. 9(8):1735–1780

    1997

  6. [2013]

    Mnih, V .; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. A. 2013. Playing atari with deep reinforcement learning. CoRR abs/1312.5602

    2013 arXiv

  7. [2014]

    Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In In EMNLP

    2014

  8. [2015]

    He, J.; Chen, J.; He, X.; Gao, J.; Li, L.; Deng, L.; and Ostendorf, M. 2015. Deep reinforcement learning with an unbounded action space. CoRR abs/1511.04636

    2015 arXiv

  9. [2016]

    P.; Mirza, M.; Graves, A.; Lil- licrap, T

    Mnih, V .; Badia, A. P.; Mirza, M.; Graves, A.; Lil- licrap, T. P.; Harley, T.; Silver, D.; and Kavukcuoglu, K

  10. [2017]

    Fulda, N.; Ricks, D.; Murdoch, B.; and Wingate, D

  11. [2018]

    Ammanabrolu, P., and Riedl, M. O. 2018. Playing text-adventure games with graph-based deep reinforcement learning. CoRR abs/1812.01628

    2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.