CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World

Aleksandr I. Panov; Alexey Skrynnik; Gregory Gorbov; Petr Kuderov; Zoya Volovikova

REVIEW 4 major objections 5 minor 14 references

CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CrafText is a benchmark that measures whether agents can follow natural-language instructions in a dynamic, changing world, with 3,924 instructions and a dual evaluation protocol for generalization.

desk verdict CrafText ships a usable dynamic instruction-following benchmark with a genuinely new evaluation split, but unvalidated checkers and dataset inconsistencies mean the reported numbers are calibration, not proof. read the letter →

arxiv 2505.11962 v1 pith:ET5XHM36 submitted 2025-05-17 cs.AI

Zoya Volovikova , Gregory Gorbov , Petr Kuderov , Aleksandr I. Panov , Alexey Skrynnik This is my paper · ORCID

classification cs.AI

keywords instructionfollowingmultimodalbenchmarkreinforcementlearningdynamicenvironmentslanguagegroundinggeneralizationopen-endedworldCrafText

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CrafText is a benchmark that asks an agent to follow natural-language instructions inside Craftax, a dynamic, procedurally generated game world where objects move and change independently of the agent. The paper builds a dataset of 3,924 instructions with 3,423 unique words, spanning Localization, Conditional, Building, and Achievement tasks, each paired with a hand-written function that checks whether the goal was met. It also proposes a dual evaluation protocol that measures generalization to paraphrased instructions and to new combinations of familiar objects. Reported baselines achieve only 0.15–0.45 success on the training split, with Dynalang nearly failing, which calibrates the benchmark as hard for current methods. If the benchmark is valid, it gives the field a reusable test of linguistic flexibility and adaptive decision-making in one setting.

What carries the argument

The load-bearing mechanism is the scenario checker: a hand-written function, parameterized by goal-specific arguments such as block type, shape, position, and ordering, that encodes the semantics of an instruction class and verifies completion from the world state at every timestep. Each of the twelve scenario templates (build, localization, conditional, achievement, and combinations) becomes many goals by enumerating parameters; GPT-4 only generates surface language and the call syntax for the checker, never the task logic. The checker doubles as the episode-termination condition and the reward signal, so the entire evaluation reduces to whether these functions faithfully capture what the instructions mean. The Craftax base supplies the dynamic world, with stochastic transitions, autonomous entities, and open-ended crafting, making the checker the fixed point the agent must satisfy under changing conditions.

What would settle it

Take a random sample of episodes across all four task categories, record the agent's trajectory, and have independent human annotators decide from the instruction text alone whether the agent satisfied the goal; compare their verdicts to the checker functions' verdicts. Substantial disagreement, such as checkers accepting clearly wrong constructions or rejecting clearly satisfied orders, would falsify the claim that success rate measures instruction following.

Watch

Extended reading notes

Core claim

The paper's central claim is that CrafText can evaluate instruction following under conditions that combine linguistic diversity with environmental dynamics, something existing benchmarks keep separate. The discovery, on the paper's own terms, is the measurement protocol: each of 496 goals (203 held out) is generated from twelve reusable scenario templates, phrased by GPT-4 into roughly six natural-language variants, and verified by scenario checker functions that run every timestep inside the JAX-accelerated Craftax environment. The benchmark shows that a planning-augmented PPO variant (PPO-T+) reaches a 0.45 training success rate and 0.28 on novel object combinations, while the world-model method Dynalang, strong in simpler grounded-language settings, falls to 0.15 training and 0.10 on new objects. The authors use these results to argue that preprocessing instructions into structured plans helps agents decompose novel goals, and that the benchmark exposes a gap between static-environment competence and dynamic instruction following.

Load-bearing premise

The load-bearing premise is that the hand-written scenario checker functions correctly and completely encode the meaning of every GPT-4-generated instruction; if any checker is too strict, too loose, or mismatched to what the words actually ask, the reported success rates stop measuring instruction following.

Editorial extensions

If this is right

Researchers can now measure linguistic sensitivity separately from environmental sensitivity: the Paraphrased split isolates wording effects, while the New Objects split isolates recombination of familiar skills.
Planning-based preprocessing (PPO-T+) outperforms direct embedding (PPO-T and FiLM) on unseen object combinations, suggesting that converting instructions into explicit subgoal plans is a promising direction for instruction following in dynamic worlds.
A strong world-model agent such as Dynalang can score near zero in this benchmark even after substantial training, meaning static-environment competence does not automatically transfer to dynamic instruction following.
Because the checker runs per timestep and is parameterized, new tasks can be added by writing one template and enumerating parameters, so the benchmark can grow beyond its initial 496 goals without recollecting data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the checkers are sound, the benchmark's hardest findings are not the headline numbers but the failure gradient by object: tasks involving water, iron, coal, and plants approach zero success, which points to perception and inventory-state bottlenecks rather than language understanding as the next limiting factor.
The paper's stated limitation that all instructions are generated by GPT-4 without human input means the linguistic range is bounded by that model's writing style; replacing a subset with human paraphrases would test whether the reported rankings survive more natural phrasing.
Because the dataset exposes the function-call format for each instruction, one can turn CrafText into a closed-loop test for LLM planners by using the checker as a verifier, effectively making it an environment for evaluating agentic planning rather than only end-to-end RL.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Zoya—quick take on arXiv:2505.11962. CrafText is a genuinely useful benchmark artifact: it puts a large-vocabulary instruction-following dataset into Craftax, giving you dynamic transitions, four task categories, and a dual evaluation split for paraphrase and new-object generalization. That combination is new as far as the cited literature goes, and the open-source JAX implementation plus baseline numbers (PPO variants, FiLM, Dynalang, zero-shot LLMs) make it immediately usable as a difficulty calibration tool. The paper is honest about its main limitation—no human-generated instructions—and that counts for something.

The soft spots are real but not fatal. The biggest one is the scenario checkers. They were hand-written by one person with no independent validation, and Appendix D's prompt asks GPT-4 to emit a check_lambda alongside each instruction, which blurs the line between language generation and task verification. If a checker is too loose or too strict, the success rates in Table 2 don't measure instruction following. That risk deserves a sensitivity analysis: spot-check terminal states, have a second annotator reimplement a subset of checkers, report false positives and false negatives. Without that, the numbers are calibration rather than rigorous evaluation.

The dataset description also has internal inconsistencies that suggest the pipeline wasn't audited: the abstract says 3,423 unique words, Section 4.1 says 2,923; goal counts sum to 596 in the complexity breakdown when the total is stated as 496; Figure 3 says 162 goals and 972 instructions. These are easy to fix but need a careful pass before publication.

The evaluation itself is thin in places—no confidence intervals or variance across seeds reported in the main table, and the training protocol is limited to the Medium split—but for a benchmark paper that's acceptable; you're mainly providing a yardstick, not a state-of-the-art comparison.

Bottom line: worth a serious referee. The benchmark is concrete, reproducible, and fills a real gap. The fixes are straightforward—validate checkers, reconcile the counts, tighten the writing. I'd take it as a conditional accept if the authors address the checker validation head-on.

Referee Report

4 major / 5 minor

Summary. CrafText is a benchmark paper that extends the Craftax environment with a dataset of 3,924 natural-language instructions, 12 scenario checkers, and four task categories (Localization, Conditional, Building, and Achievement). The authors propose a dual evaluation protocol: a Paraphrased test set for linguistic generalization and a New Objects test set for generalization to novel combinations of familiar objects. They implement the benchmark in JAX with XLA acceleration and report baseline results for PPO-T, PPO-T+, FiLM, and Dynalang, finding that all methods perform well below ceiling and that planning-augmented PPO-T+ generalizes best to new object combinations. The central claim is that CrafText provides a reusable, dynamic, high-vocabulary benchmark for instruction following, with paired goal checkers enabling automatic verification.

Significance. If the benchmark is valid, it fills a real gap: it combines a dynamic open-ended environment, relatively large instruction vocabulary, and an evaluation protocol that separates linguistic paraphrase generalization from compositional goal generalization. The open-source JAX implementation and the inclusion of several strong baselines are concrete assets, and the dual evaluation protocol is a useful methodological step for the instruction-following community. The reported speed numbers and baseline calibration are also useful for future work. However, the significance is currently conditional on the correctness and completeness of the hand-written scenario checkers and on the internal consistency of the dataset statistics; both need to be established before the benchmark can be relied upon. The absence of independent checker validation and the unresolved numerical inconsistencies in the dataset description are load-bearing issues that affect every success-rate number in the paper.

major comments (4)

[Section 4.1, Section 4.2, Appendix D, Table 2] The central measurement in the paper is success rate computed by scenario checker functions, but these checkers are not validated. Section 4.1 states that all scenario verification functions were implemented by one person familiar with Craftax, yet no inter-annotator agreement, no human-labeled terminal states, no false-positive/false-negative analysis, and no independent reimplementation is reported. If a checker is too loose, an agent can receive success without satisfying the instruction; if too strict, correct behavior is scored as failure. In either case, the Table 2 SR values do not reliably measure instruction following. The concern is amplified by the instruction-generation prompt in Appendix D, which asks GPT-4 to emit a `check_lambda` alongside each instruction, blurring the claimed separation in Section 4.2 between GPT-4 as a purely linguistic generator and the human-written verification logic. The authors need to provide external validation of the checkers, for example through human-labeled episodes and a separate checker implementation, before the reported success rates can be interpreted as measuring instruction following.
[Abstract, Section 4.1, Figure 3, Table 3] The dataset statistics are internally inconsistent, and this undermines reproducibility. The abstract reports 3,423 unique words, while Section 4.1 reports 2,923. Section 4.1 states there are 496 goals, split as 100 Easy, 277 Medium, and 219 Hard, but 100 + 277 + 219 = 596, not 496. Figure 3 states the dataset features 162 goals and 972 instructions, which is incompatible with the claimed 496 goals and 3,924 instructions. Table 3 sums to 496 implemented goals, so the discrepancy is not a simple arithmetic slip in the table but rather in the text and figure. The authors should audit the generation pipeline and reconcile all counts, because the dataset size and difficulty distribution are part of the benchmark's advertised properties and feed directly into how users interpret the experimental results.
[Section 4.2, Section 5.2, Section 7] The Paraphrased test set does not provide an independent measure of linguistic generalization. Both training instructions and the paraphrased test instructions are generated by the same GPT-4 pipeline with the same prompt template, as described in Section 4.2 and Appendix D, and Section 7 acknowledges the absence of human-generated instructions. Consequently, a model that overfits to GPT-4's linguistic distribution will look better on both splits, and the observed drops in Section 5.2 may underestimate generalization difficulty relative to human language variation. The claim that the protocol measures 'novel instruction formulations' should be qualified to 'novel relative to a single LLM generator.' A concrete remedy would be to add a small human-paraphrased test set or a second independent generator, and to report success rates on it separately.
[Section 5.1, Table 2] The experimental comparison would be considerably stronger with error bars or variance information. Table 2 states that results are computed across 50 seeds, but only point estimates are reported. Given that the main findings include small differences among PPO-T (0.40), PPO-T+ (0.45), and FiLM (0.43) on the training set, and smaller differences on the test sets, the absence of any measure of variance makes it impossible to judge whether the reported ordering is reliable. Adding standard deviations or confidence intervals, at least for the total rows, is necessary for the baseline calibration to be useful to subsequent users of the benchmark.

minor comments (5)

[Introduction] There are typos and infelicities in the introductory text, such as 'amount objects and way to interact' and 'ADonemarker' in the Figure 1 caption; these should be corrected in a revision.
[Introduction, References] The citation for CLIP is given as (Yao et al., 2022) in the introduction, but the reference list identifies Yao et al. with DetCLIP and attributes CLIP to Radford et al. (2021); the in-text citation should be corrected.
[Section 5, Appendix I] The naming of baselines is inconsistent: the main text uses PPO-T and PPO-T+, while Figure 14 and Appendix I use PPO-B and PPO-B-GPT4Plans; the notation should be unified throughout the paper.
[Section 4.3] The sentence 'The both possible observation types (visual and vector-based) of Craftax are augmented with instructions' contains a grammatical error; it should read 'Both possible observation types...'.
[Appendix D] The generation prompt includes a field named `check_lambda` with a lambda expression. Even if the lambda only calls a human-written scenario function, the name and format invite confusion about whether GPT-4 is contributing verification logic; the appendix should clarify that the lambda is generated as a function call and never defines the checking logic itself.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the benchmark construction and baseline evaluations are self-contained.

full rationale

CrafText is a benchmark-construction paper rather than a mathematical derivation, so the main circularity patterns do not apply. The scenario checker functions are hand-written by a human familiar with Craftax and are not defined in terms of the measured success rates, so there is no self-definitional reduction. GPT-4 generates both the training instructions and the paraphrased test variants, but the paper's Section 7 explicitly acknowledges the absence of human-generated instructions as a limitation; this weakens external validity but does not make the held-out paraphrase split equivalent to the training distribution by construction. The only overlapping self-citations (e.g., the IGLU reference) appear in related-work comparisons and are not load-bearing for any claim. The reported baselines are direct measurements of success rate, not fitted parameters relabeled as predictions. Internal inconsistencies in dataset statistics (vocabulary 3,423 vs 2,923; goal counts 100/277/219 vs Table 3) are correctness and auditing concerns, not circularity. No equation or claim in the paper reduces to its own input.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of hand-written checkers, the representativeness of GPT-4-generated language, the suitability of Craftax as a dynamic-world proxy, and the representativeness of the baselines. None of these premises is independently verified with external evidence in the paper.

assumptions (4)

domain assumption Scenario checker functions faithfully encode goal completion for all 496 goals.
Section 4.1 says checkers were implemented by one person familiar with Craftax; no independent validation, human agreement study, or oracle experiments are reported.
domain assumption GPT-4-generated instructions and paraphrases approximate the linguistic diversity of real human instructions.
Section 4.2 uses GPT-4 for all instruction generation; Section 7 explicitly lists the absence of human-generated instructions as a limitation.
domain assumption Craftax is a valid open-ended dynamic environment for instruction-following research.
Section 4 builds on Craftax and inherits its world dynamics; if Craftax is not representative of real-world volatility, the benchmark's intended transfer is questionable.
domain assumption The selected baselines (PPO with DistilBERT, FiLM, and Dynalang) are sufficient to characterize benchmark difficulty.
Section 5.1 uses these baselines to argue that current methods struggle; the conclusion depends on these methods being representative of the field.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World." pith.science (2026). https://pith.science/paper/ET5XHM36

@misc{pith2026250511962,
  author       = {Pith},
  title        = {Pith review of: CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ET5XHM36}},
  note         = {Machine review of arXiv:2505.11962}
}

read the original abstract

Following instructions in real-world conditions requires the ability to adapt to the world's volatility and entanglement: the environment is dynamic and unpredictable, instructions can be linguistically complex with diverse vocabulary, and the number of possible goals an agent may encounter is vast. Despite extensive research in this area, most studies are conducted in static environments with simple instructions and a limited vocabulary, making it difficult to assess agent performance in more diverse and challenging settings. To address this gap, we introduce CrafText, a benchmark for evaluating instruction following in a multimodal environment with diverse instructions and dynamic interactions. CrafText includes 3,924 instructions with 3,423 unique words, covering Localization, Conditional, Building, and Achievement tasks. Additionally, we propose an evaluation protocol that measures an agent's ability to generalize to novel instruction formulations and dynamically evolving task configurations, providing a rigorous test of both linguistic understanding and adaptive decision-making.

Figures

Figures reproduced from arXiv: 2505.11962 by the authors.

**Figure 2.** The figure illustrates the hierarchical structure [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Left: Data Gathering Pipeline – experts define goal templates expanded with GPT to generate tasks, instructions, and goal-checking functions (e.g., build a square with stones or plants). Middle: CrafText Dataset – features 162 goals and 972 instructions (162 × 6), combining scenario checkers, goals, and instructions with varied parameters like block type and size. Right: Interactive Environment – the agent follows i… view at source ↗

Figures from the paper (11 more)

**Figure 7.** Figure 7: Achievements Tasks This type of task evaluates the agent’s ability to understand achievements by interpreting tasks and identifying the necessary actions. It also tests whether the agent can combine multiple achievements in sequence or simultaneously. Additionally, it…

**Figure 5.** Figure 5: Building Instruction Example This category tests the agent’s spatial reasoning and ability to translate instructions into precise constructions (see an Example at [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Localization Instruction Example In this example, the agent needs to: 1) Identify the lake 2) Determine the left side 3) Understand relative positioning Additionally, these tasks may involve directional terms such as left, right, above, below, north, south, west, and e…

**Figure 8.** Figure 8: Examples of instructions in dataset. C DATASET: Per Category Scenarios, Goal, Instructions In our dataset, there is a division into the training set, the test set with rephrased instructions (Test Paraphrased), and the test set with new tasks (Test [PITH_FULL_IMAGE:fi…

**Figure 9.** Figure 9: Example instruction set for different tasks. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 12.** Figure 12: Examples of the visual observation input in [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

**Figure 10.** Figure 10: SPS of CrafText environment based on Craftax Classic environment 128 256 512 1024 2048 NUM ENVS 8000 12000 16000 20000 24000 SPS (Steps Per Second) CrafText (Craftax) Tesla V100 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: SPS of CrafText environment based on Craftax environment [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

**Figure 13.** Figure 13: Visualizations of multimodal environments. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

**Figure 14.** Figure 14: The training and corresponding test (other parameters) curves for three baselines (PPO+BertEmb, [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

**Figure 15.** Figure 15: Aggregated SR for tasks involving different [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages

[1]

When specifying target objects (objects with which the player will interact), use different syn- onyms in paraphrases to assess the vocabulary range of player 2

work page
[2]

InProceedings of the NeurIPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pages 204–216

Interactive grounded language understanding in a collaborative environment: Retrospective on iglu 2022 competition. InProceedings of the NeurIPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pages 204–216. PMLR. Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan...

work page 2022
[3]

In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10965– 10975

Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10965– 10975. Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, P. Abbeel, Dan Klein, and Anca D. Dragan. 2023. Learning to model the world with language.ArXiv, abs/2308.01399. Corey Lynch, Ayzaan Wahid, Jonathan Tompson,...

arXiv 2023
[4]

After collecting coal, the player should gather wood and then place a stone on the crafting table

Dino: Detr with improved denoising anchor boxes for end-to-end object detection.arXiv preprint arXiv:2203.03605. Victor Zhong, Tim Rocktäschel, and Edward Grefen- stette. 2019. Rtfm: Generalising to novel envi- ronment dynamics via reading.arXiv preprint arXiv:1910.08210. A DA TASET: Tasks Description Conditional. In theConditionalcategory, the agent is r...

arXiv 2019
[6]

Present the target objects in varying orders to evaluate how well player 2 understands different language structures

work page
[7]

For each set of paraphrases, sort them from the simplest language to the most complex

work page
[8]

Example instruction here

Ensure the instructions are as varied as possi- ble with a broad vocabulary. Format your answer as a Python dictionary with the following structure: 1instructions = { 2instruction_id: { 3’instruction’: "Example instruction here", 4’instruction_paraphrases’: [ 5"Paraphrase 1 here", 6"Paraphrase 2 here", 7"Paraphrase 3 here", 8"Paraphrase 4 here", 9"Paraphr...

work page 2022
[9]

Using only object names existing in the Craftax environment, provide the plan as a numbered list

work page

Show all 14 references

[10]

Each step should outline a specific action or logical task for the agent, such as resource collection, crafting, or object placement
[11]

Check yourself! !ATTENTION!Ensure the plan uses only ob- ject names and actions existing in Craftax

Keep steps clear, concise, and implementable in Craftax, with a maximum of 5 words per step. Check yourself! !ATTENTION!Ensure the plan uses only ob- ject names and actions existing in Craftax. Replace any incorrect terms in the instruction with their correct Craftax equivalen...
[12]

Environmental Features Dynamic Instruction Following Task Setup. An agent needs to interpret the same instruction in changing visual and spatial contexts.Experimental results (Table 2):Despite extensive training, agent performance remains suboptimal even on the train- ing set....
[13]

Linguistic Features Linguistic Variation and Paraphrasing.Each goal is expressed through multiple paraphrases that differ lexically and syntactically. The agent must understand diverse formulations, recognize equiv- alence, and map them to the same behavior.Ex- perimental resu...
[14]

Make a crafting table

Generalization to Novel Combinations Test New Objects Split.Includes instructions with new combinations of objects, spatial relations, and parameters. The agent must generalize beyond memorized templates and recombine known ele- ments in novel ways.Experimental results (Table ...
[2022]

InThirty-sixth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track

Minedojo: Building open-ended embodied agents with internet-scale knowledge. InThirty-sixth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track. Jonathan Gray, Kavya Srinet, Yacine Jernite, Hao- nan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, ...

2019 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

When specifying target objects (objects with which the player will interact), use different syn- onyms in paraphrases to assess the vocabulary range of player 2

work page

[2] [2]

InProceedings of the NeurIPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pages 204–216

Interactive grounded language understanding in a collaborative environment: Retrospective on iglu 2022 competition. InProceedings of the NeurIPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pages 204–216. PMLR. Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan...

work page 2022

[3] [3]

In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10965– 10975

Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 10965– 10975. Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, P. Abbeel, Dan Klein, and Anca D. Dragan. 2023. Learning to model the world with language.ArXiv, abs/2308.01399. Corey Lynch, Ayzaan Wahid, Jonathan Tompson,...

arXiv 2023

[4] [4]

After collecting coal, the player should gather wood and then place a stone on the crafting table

Dino: Detr with improved denoising anchor boxes for end-to-end object detection.arXiv preprint arXiv:2203.03605. Victor Zhong, Tim Rocktäschel, and Edward Grefen- stette. 2019. Rtfm: Generalising to novel envi- ronment dynamics via reading.arXiv preprint arXiv:1910.08210. A DA TASET: Tasks Description Conditional. In theConditionalcategory, the agent is r...

arXiv 2019

[5] [6]

Present the target objects in varying orders to evaluate how well player 2 understands different language structures

work page

[6] [7]

For each set of paraphrases, sort them from the simplest language to the most complex

work page

[7] [8]

Example instruction here

Ensure the instructions are as varied as possi- ble with a broad vocabulary. Format your answer as a Python dictionary with the following structure: 1instructions = { 2instruction_id: { 3’instruction’: "Example instruction here", 4’instruction_paraphrases’: [ 5"Paraphrase 1 here", 6"Paraphrase 2 here", 7"Paraphrase 3 here", 8"Paraphrase 4 here", 9"Paraphr...

work page 2022

[8] [9]

Using only object names existing in the Craftax environment, provide the plan as a numbered list

work page