{"id":"5e9fdab6-7778-4732-8822-677e46f7c577","arxiv_id":"2505.11962","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CrafText, built on Craftax, contributes 3,924 instructions across four task types and a dual generalization protocol; the strongest baseline (PPO-T+) reaches only 0.45 training and 0.28 new-object success rate.","lead":"CrafText is a new benchmark for testing AI agents that follow written instructions in a dynamic, game-like world. It adds thousands of varied instructions and a test protocol that checks whether agents understand reworded commands and new object combinations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity rests on unvalidated hand-written scenario checkers; no evidence rules out checkers that are too loose or too strict.","rationale":"The strongest claim is that CrafText provides a reusable benchmark for instruction following, with baseline numbers as difficulty calibration. That claim rests on the assumption that the scenario checkers are ground-truth semantics for the instructions. The reader's weakest assumption is exactly this, and I agree. The paper gives no evidence for checker correctness beyond self-report, and the pipeline's reliance on GPT-4 for instruction generation plus the internal dataset inconsistencies show that the benchmark is not yet at the level of validation needed to support the central claim. I do not see a more fundamental objection: the environment is genuinely dynamic, the vocabulary is large, and the dual test split is a reasonable protocol if the checkers are sound. Because the missing validation is fixable by an audit and does not by itself disprove the benchmark's utility, the correct verdict remains CONDITIONAL rather than REJECT.","tokens_in":17766,"tokens_out":5562,"duration_ms":57645,"concrete_test":"Audit the checkers directly: sample 100 instructions stratified by scenario and difficulty; ask two independent Craftax experts, blind to the original checkers, to write verification functions purely from the instruction text and environment API. Run the original and both new checkers on a fixed corpus of 1,000 agent trajectories (including near-misses and deliberately wrong actions), and compare outcomes. Additionally, have human annotators label 200 terminal states as success/failure against the instruction. If pairwise checker agreement is below 95% or human-checker agreement is below 95%, the SR metric is not trustworthy and the benchmark should be released with corrected checkers; if agreement is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central measurement—success rate—is only meaningful if the hand-written scenario checkers (Section 4.1: 'All scenario verification functions were implemented by a person familiar with the mechanics of the Craftax environment') exactly and completely encode the semantic content of the GPT-4-generated instructions. The paper reports no validation of these checkers: no inter-annotator agreement, no human-labeled terminal states, no false-positive/false-negative analysis. Appendix D's prompt even asks GPT-4 to emit a `check_lambda` alongside each instruction, so the boundary between 'language generator' (Section 4.2) and task verification is not clearly enforced. If a checker is too loose, an agent can receive success without performing the instructed action; if too strict, correct behavior is scored as failure. Either way, the reported SR values (Table 2) do not measure instruction following. This risk is amplified by internal inconsistencies in the dataset description (vocabulary 3,423 in the abstract vs 2,923 in Section 4.1; 496 goals vs 596 when complexity counts are summed; Figure 3 says 162 goals/972 instructions), which suggest the data-generation pipeline has not been carefully audited.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CrafText is a benchmark paper that extends the Craftax environment with a dataset of 3,924 natural-language instructions, 12 scenario checkers, and four task categories (Localization, Conditional, Building, and Achievement). The authors propose a dual evaluation protocol: a Paraphrased test set for linguistic generalization and a New Objects test set for generalization to novel combinations of familiar objects. They implement the benchmark in JAX with XLA acceleration and report baseline results for PPO-T, PPO-T+, FiLM, and Dynalang, finding that all methods perform well below ceiling and that planning-augmented PPO-T+ generalizes best to new object combinations. The central claim is that CrafText provides a reusable, dynamic, high-vocabulary benchmark for instruction following, with paired goal checkers enabling automatic verification.","tokens_in":18004,"tokens_out":4835,"duration_ms":51655,"significance":"If the benchmark is valid, it fills a real gap: it combines a dynamic open-ended environment, relatively large instruction vocabulary, and an evaluation protocol that separates linguistic paraphrase generalization from compositional goal generalization. The open-source JAX implementation and the inclusion of several strong baselines are concrete assets, and the dual evaluation protocol is a useful methodological step for the instruction-following community. The reported speed numbers and baseline calibration are also useful for future work. However, the significance is currently conditional on the correctness and completeness of the hand-written scenario checkers and on the internal consistency of the dataset statistics; both need to be established before the benchmark can be relied upon. The absence of independent checker validation and the unresolved numerical inconsistencies in the dataset description are load-bearing issues that affect every success-rate number in the paper.","major_comments":[{"comment":"The central measurement in the paper is success rate computed by scenario checker functions, but these checkers are not validated. Section 4.1 states that all scenario verification functions were implemented by one person familiar with Craftax, yet no inter-annotator agreement, no human-labeled terminal states, no false-positive/false-negative analysis, and no independent reimplementation is reported. If a checker is too loose, an agent can receive success without satisfying the instruction; if too strict, correct behavior is scored as failure. In either case, the Table 2 SR values do not reliably measure instruction following. The concern is amplified by the instruction-generation prompt in Appendix D, which asks GPT-4 to emit a `check_lambda` alongside each instruction, blurring the claimed separation in Section 4.2 between GPT-4 as a purely linguistic generator and the human-written verification logic. The authors need to provide external validation of the checkers, for example through human-labeled episodes and a separate checker implementation, before the reported success rates can be interpreted as measuring instruction following.","section":"Section 4.1, Section 4.2, Appendix D, Table 2"},{"comment":"The dataset statistics are internally inconsistent, and this undermines reproducibility. The abstract reports 3,423 unique words, while Section 4.1 reports 2,923. Section 4.1 states there are 496 goals, split as 100 Easy, 277 Medium, and 219 Hard, but 100 + 277 + 219 = 596, not 496. Figure 3 states the dataset features 162 goals and 972 instructions, which is incompatible with the claimed 496 goals and 3,924 instructions. Table 3 sums to 496 implemented goals, so the discrepancy is not a simple arithmetic slip in the table but rather in the text and figure. The authors should audit the generation pipeline and reconcile all counts, because the dataset size and difficulty distribution are part of the benchmark's advertised properties and feed directly into how users interpret the experimental results.","section":"Abstract, Section 4.1, Figure 3, Table 3"},{"comment":"The Paraphrased test set does not provide an independent measure of linguistic generalization. Both training instructions and the paraphrased test instructions are generated by the same GPT-4 pipeline with the same prompt template, as described in Section 4.2 and Appendix D, and Section 7 acknowledges the absence of human-generated instructions. Consequently, a model that overfits to GPT-4's linguistic distribution will look better on both splits, and the observed drops in Section 5.2 may underestimate generalization difficulty relative to human language variation. The claim that the protocol measures 'novel instruction formulations' should be qualified to 'novel relative to a single LLM generator.' A concrete remedy would be to add a small human-paraphrased test set or a second independent generator, and to report success rates on it separately.","section":"Section 4.2, Section 5.2, Section 7"},{"comment":"The experimental comparison would be considerably stronger with error bars or variance information. Table 2 states that results are computed across 50 seeds, but only point estimates are reported. Given that the main findings include small differences among PPO-T (0.40), PPO-T+ (0.45), and FiLM (0.43) on the training set, and smaller differences on the test sets, the absence of any measure of variance makes it impossible to judge whether the reported ordering is reliable. Adding standard deviations or confidence intervals, at least for the total rows, is necessary for the baseline calibration to be useful to subsequent users of the benchmark.","section":"Section 5.1, Table 2"}],"minor_comments":[{"comment":"There are typos and infelicities in the introductory text, such as 'amount objects and way to interact' and 'ADonemarker' in the Figure 1 caption; these should be corrected in a revision.","section":"Introduction"},{"comment":"The citation for CLIP is given as (Yao et al., 2022) in the introduction, but the reference list identifies Yao et al. with DetCLIP and attributes CLIP to Radford et al. (2021); the in-text citation should be corrected.","section":"Introduction, References"},{"comment":"The naming of baselines is inconsistent: the main text uses PPO-T and PPO-T+, while Figure 14 and Appendix I use PPO-B and PPO-B-GPT4Plans; the notation should be unified throughout the paper.","section":"Section 5, Appendix I"},{"comment":"The sentence 'The both possible observation types (visual and vector-based) of Craftax are augmented with instructions' contains a grammatical error; it should read 'Both possible observation types...'.","section":"Section 4.3"},{"comment":"The generation prompt includes a field named `check_lambda` with a lambda expression. Even if the lambda only calls a human-written scenario function, the name and format invite confusion about whether GPT-4 is contributing verification logic; the appendix should clarify that the lambda is generated as a function call and never defines the checking logic itself.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially valuable, but the unvalidated scenario checkers and the inconsistent dataset counts are serious enough that I would not accept the paper in its current form. I do not see these as irreparable flaws: the checker issue can be addressed with human-labeled states, false-positive/false-negative analysis, and independent reimplementation, and the statistical inconsistencies can be fixed by auditing the pipeline. I would also ask the editor to ensure that the generation and validation artifacts are made available along with the revision, since the paper's central claims depend on them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zoya—quick take on arXiv:2505.11962. CrafText is a genuinely useful benchmark artifact: it puts a large-vocabulary instruction-following dataset into Craftax, giving you dynamic transitions, four task categories, and a dual evaluation split for paraphrase and new-object generalization. That combination is new as far as the cited literature goes, and the open-source JAX implementation plus baseline numbers (PPO variants, FiLM, Dynalang, zero-shot LLMs) make it immediately usable as a difficulty calibration tool. The paper is honest about its main limitation—no human-generated instructions—and that counts for something.\n\nThe soft spots are real but not fatal. The biggest one is the scenario checkers. They were hand-written by one person with no independent validation, and Appendix D's prompt asks GPT-4 to emit a check_lambda alongside each instruction, which blurs the line between language generation and task verification. If a checker is too loose or too strict, the success rates in Table 2 don't measure instruction following. That risk deserves a sensitivity analysis: spot-check terminal states, have a second annotator reimplement a subset of checkers, report false positives and false negatives. Without that, the numbers are calibration rather than rigorous evaluation.\n\nThe dataset description also has internal inconsistencies that suggest the pipeline wasn't audited: the abstract says 3,423 unique words, Section 4.1 says 2,923; goal counts sum to 596 in the complexity breakdown when the total is stated as 496; Figure 3 says 162 goals and 972 instructions. These are easy to fix but need a careful pass before publication.\n\nThe evaluation itself is thin in places—no confidence intervals or variance across seeds reported in the main table, and the training protocol is limited to the Medium split—but for a benchmark paper that's acceptable; you're mainly providing a yardstick, not a state-of-the-art comparison.\n\nBottom line: worth a serious referee. The benchmark is concrete, reproducible, and fills a real gap. The fixes are straightforward—validate checkers, reconcile the counts, tighten the writing. I'd take it as a conditional accept if the authors address the checker validation head-on.","headline":"CrafText ships a usable dynamic instruction-following benchmark with a genuinely new evaluation split, but unvalidated checkers and dataset inconsistencies mean the reported numbers are calibration, not proof.","tokens_in":18495,"tokens_out":2205,"would_cite":false,"duration_ms":21636,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CrafText is a benchmark that measures whether agents can follow natural-language instructions in a dynamic, changing world, with 3,924 instructions and a dual evaluation protocol for generalization.","keywords":["instruction following","multimodal benchmark","reinforcement learning","dynamic environments","language grounding","generalization","open-ended world","CrafText"],"falsifier":"Take a random sample of episodes across all four task categories, record the agent's trajectory, and have independent human annotators decide from the instruction text alone whether the agent satisfied the goal; compare their verdicts to the checker functions' verdicts. Substantial disagreement, such as checkers accepting clearly wrong constructions or rejecting clearly satisfied orders, would falsify the claim that success rate measures instruction following.","tokens_in":17629,"feed_emoji":"🧭","tokens_out":5810,"duration_ms":55251,"temperature":0.7,"pith_summary":"CrafText is a benchmark that asks an agent to follow natural-language instructions inside Craftax, a dynamic, procedurally generated game world where objects move and change independently of the agent. The paper builds a dataset of 3,924 instructions with 3,423 unique words, spanning Localization, Conditional, Building, and Achievement tasks, each paired with a hand-written function that checks whether the goal was met. It also proposes a dual evaluation protocol that measures generalization to paraphrased instructions and to new combinations of familiar objects. Reported baselines achieve only 0.15–0.45 success on the training split, with Dynalang nearly failing, which calibrates the benchmark as hard for current methods. If the benchmark is valid, it gives the field a reusable test of linguistic flexibility and adaptive decision-making in one setting.","feed_headline":"3,924 instructions test agents that must obey orders in a changing world","feed_subtitle":"Agents that first turn instructions into plans generalize best to unseen object combinations, first results show.","key_machinery":"The load-bearing mechanism is the scenario checker: a hand-written function, parameterized by goal-specific arguments such as block type, shape, position, and ordering, that encodes the semantics of an instruction class and verifies completion from the world state at every timestep. Each of the twelve scenario templates (build, localization, conditional, achievement, and combinations) becomes many goals by enumerating parameters; GPT-4 only generates surface language and the call syntax for the checker, never the task logic. The checker doubles as the episode-termination condition and the reward signal, so the entire evaluation reduces to whether these functions faithfully capture what the instructions mean. The Craftax base supplies the dynamic world, with stochastic transitions, autonomous entities, and open-ended crafting, making the checker the fixed point the agent must satisfy under changing conditions.","core_discovery":"The paper's central claim is that CrafText can evaluate instruction following under conditions that combine linguistic diversity with environmental dynamics, something existing benchmarks keep separate. The discovery, on the paper's own terms, is the measurement protocol: each of 496 goals (203 held out) is generated from twelve reusable scenario templates, phrased by GPT-4 into roughly six natural-language variants, and verified by scenario checker functions that run every timestep inside the JAX-accelerated Craftax environment. The benchmark shows that a planning-augmented PPO variant (PPO-T+) reaches a 0.45 training success rate and 0.28 on novel object combinations, while the world-model method Dynalang, strong in simpler grounded-language settings, falls to 0.15 training and 0.10 on new objects. The authors use these results to argue that preprocessing instructions into structured plans helps agents decompose novel goals, and that the benchmark exposes a gap between static-environment competence and dynamic instruction following.","pith_inferences":["If the checkers are sound, the benchmark's hardest findings are not the headline numbers but the failure gradient by object: tasks involving water, iron, coal, and plants approach zero success, which points to perception and inventory-state bottlenecks rather than language understanding as the next limiting factor.","The paper's stated limitation that all instructions are generated by GPT-4 without human input means the linguistic range is bounded by that model's writing style; replacing a subset with human paraphrases would test whether the reported rankings survive more natural phrasing.","Because the dataset exposes the function-call format for each instruction, one can turn CrafText into a closed-loop test for LLM planners by using the checker as a verifier, effectively making it an environment for evaluating agentic planning rather than only end-to-end RL."],"forward_implications":["Researchers can now measure linguistic sensitivity separately from environmental sensitivity: the Paraphrased split isolates wording effects, while the New Objects split isolates recombination of familiar skills.","Planning-based preprocessing (PPO-T+) outperforms direct embedding (PPO-T and FiLM) on unseen object combinations, suggesting that converting instructions into explicit subgoal plans is a promising direction for instruction following in dynamic worlds.","A strong world-model agent such as Dynalang can score near zero in this benchmark even after substantial training, meaning static-environment competence does not automatically transfer to dynamic instruction following.","Because the checker runs per timestep and is parameterized, new tasks can be added by writing one template and enumerating parameters, so the benchmark can grow beyond its initial 496 goals without recollecting data."],"supporting_citations":[{"why":"Supplies the Craftax world, its dynamic mechanics, action space, and observations that CrafText extends with instructions and checkers.","marker":"(Matthews et al., 2024)"},{"why":"Provides Dynalang, the model-based baseline whose near-zero success rate the paper uses to show that static-world methods struggle here.","marker":"(Lin et al., 2023)"},{"why":"Provides PPO, the optimization algorithm underlying the PPO-T, PPO-T+, and FiLM baselines.","marker":"(Schulman et al., 2017)"},{"why":"Provides FiLM conditioning layers, one of the baseline methods that modulates visual features by instruction text.","marker":"(Perez et al., 2018)"},{"why":"Defines Crafter, the parent open-world survival environment whose mechanics Craftax and therefore CrafText inherit.","marker":"(Hafner, 2022)"}],"fun_headline_variants":["CrafText: dynamic benchmark tests agent instruction following","Plan-first agents beat world models on CrafText dynamic tasks","New benchmark: 3,924 instructions in a changing world","CrafText exposes agent gap: planning wins over adaptation","Benchmark measures instruction following under volatility"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-written scenario checker functions correctly and completely encode the meaning of every GPT-4-generated instruction; if any checker is too strict, too loose, or mismatched to what the words actually ask, the reported success rates stop measuring instruction following.","fun_headline_variants_meta":{"raw":{"variants":["CrafText: dynamic benchmark tests agent instruction following","Plan-first agents beat world models on CrafText dynamic tasks","New benchmark: 3,924 instructions in a changing world","CrafText exposes agent gap: planning wins over adaptation","Benchmark measures instruction following under volatility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1322,"prompt_tokens":897,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":513,"tokens_out":425,"duration_ms":4752,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:43:23.662375+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of episodes across all four task categories, record the agent's trajectory, and have independent human annotators decide from the instruction text alone whether the agent satisfied the goal; compare their verdicts to the checker functions' verdicts. Substantial disagreement, such as checkers accepting clearly wrong constructions or rejecting clearly satisfied orders, would falsify the claim that success rate measures instruction following.","supporting_citations":[],"review_version":1}