{"id":"684d9093-0ec2-46e6-b7c6-54b33ea3dd2a","arxiv_id":"2505.20726","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"ManiTaskGen automatically generates diverse, feasible mobile manipulation tasks from any input scene, and uses them to benchmark and improve vision-language robot agents.","lead":"This paper introduces ManiTaskGen, a system that automatically turns a 3D scene into tens of thousands of feasible robot pick-and-place tasks, with both concrete instructions like 'move the mug to the table' and abstract ones like 'clear the table'. The main use is generating large benchmarks and training exercises for vision-language robot agents without hand-writing tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The near-exhaustiveness claim is scoped to navigate/pick/place, but the paper states it broadly; non-pose tasks (turn on, open, pour) are excluded by the formalization, so the central claim overreaches.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the formalization of scene state and atomic actions excludes non-pose state changes and finer manipulation skills, so the generated task set is not near-exhaustive for realistic scenes. The paper's own limitations section confirms this exclusion, making the concern concrete rather than speculative. I agree with the reader that this does not invalidate the system as a pick-and-place task generator, but it does undermine the universal and near-exhaustive claims as stated in the abstract and introduction. The appropriate remedy is to condition the claims explicitly on the navigate/pick/place action set, which is a wording/scope adjustment rather than a change to the core contribution. Therefore the reader's CONDITIONAL verdict remains appropriate, and my analysis does not move it. Other weaknesses noted by the reader (lack of code release, small sample sizes, no Level-4 evaluation) are secondary and do not change the central assessment. I see no internal inconsistency in the formalization itself; the issue is that the paper asserts a stronger conclusion than its premises support. A concrete test comparing generated tasks against a curated list containing non-pose tasks would settle whether the claim is meant broadly (and false) or narrowly (and understated).","tokens_in":56917,"tokens_out":4507,"duration_ms":49054,"concrete_test":"Select the ReplicaCAD apt_0 scene used in the paper and compile a set of 50 human-written feasible mobile manipulation tasks for it, deliberately including 25 tasks that require non-pose state changes (e.g., turn on a lamp, open a drawer, pour from a cup, toggle a switch, cut an object) and 25 pure pick-and-place tasks. Run the ManiTaskGen pipeline exactly as described (scene graph construction, atomic-action enumeration, process/outcome generation) on the same scene. Count how many of the 50 tasks are generated verbatim or can be generated by instantiating the provided templates and action patterns.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that ManiTaskGen generates a 'comprehensive and diverse collection of feasible mobile manipulation tasks that are logically near-exhaustive for that specific scene' (Sec. 1 and Sec. 3.1). The justification rests on Sec. 3.1's formalization: scene state is defined solely by object poses and containment states, atomic actions are parameterized object relocations, and the agent-environment interaction cycle 'inherently defines the space of all possible tasks.' This is only true if 'possible tasks' is implicitly restricted to tasks achievable by navigate/pick/place skills. The paper's own Limitations section (Sec. 6) concedes that non-pose state changes and fine-grained manipulation are future work. Yet the abstract and introduction do not carry this caveat: they promise 'the full spectrum of feasible tasks within any given scene' and a 'universal task generation framework for arbitrary scenes.' A scene with a lamp, a closed drawer, and a cup near a pitcher contains feasible tasks such as 'turn on the lamp,' 'open the drawer,' and 'pour water into the cup' that ManiTaskGen cannot generate because its state representation and action set cannot express them. Thus the 'inherently defines' assertion is a definitional choice, not a discovered fact; no proof is given that every mobile manipulation task reduces to object relocation. This is the most load-bearing weakness because the paper's novelty and scaling argument depend on near-exhaustiveness within a scene. Even within the pick-and-place scope, the system only samples finite sequences and discrete receptacle regions without a completeness argument, but the non-pose omission is a more fundamental boundary that directly contradicts the broad scope language.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ManiTaskGen is a system that, given scene information (object poses, bounding boxes, and optionally mesh models), automatically generates mobile manipulation task instructions. It constructs a Receptacle-Aware 3D Scene Graph, enumerates feasible atomic pick-and-place actions as object-to-receptacle relocations, chains them into process-based tasks (single- and two-step with THEN/AND/OR connectors), and instantiates 200 human-curated templates into outcome-based tasks filtered by a GPT-4o/Gemini-2.5-pro/Claude-3.7-sonnet voting ensemble. The paper evaluates the generated tasks on three scenes (ReplicaCAD, AI2THOR, SUN-RGBD), reporting human-verified validity rates of 94.0% and 86.5% for process- and outcome-based tasks versus 29.4% and 21.1% for a GPT-4o baseline; benchmarks nine VLM agents in a simulator, finding average success rates below 30% on process-based tasks; and proposes an inference-time self-reflection method that improves three VLMs. The central claim is that the generated task collection is comprehensive and logically near-exhaustive for a given scene.","tokens_in":57196,"tokens_out":14729,"duration_ms":132264,"significance":"If appropriately scoped, this is a useful contribution: it replaces manual task annotation with an automated pipeline, enables large-scale benchmarks, and provides a route to training data for embodied VLMs. Concrete strengths include: (1) the generation mechanism for process-based tasks is deterministic and transparent given the scene graph, making it reproducible and independent of the measured outcomes; (2) the human validity check is a genuine comparative result, beating the GPT-4o baseline by a large margin; (3) the benchmark harness provides automatic, precise success verification for Levels 1–3; and (4) the improvement experiment (Table 4) demonstrates concrete cross-scene transfer. The main caveat is that the “near-exhaustive” claim is only defensible within the class of tasks achievable by navigate/pick/place primitives over pose/containment states; as stated in the abstract and introduction, it overreaches. With that claim re-scoped, the framework would be a valuable tool for scaling benchmark and training-task creation across arbitrary scenes.","major_comments":[{"comment":"The paper's headline claim — that ManiTaskGen generates task collections that are “logically near-exhaustive” for a scene (Sec. 1) and cover “the full spectrum of feasible tasks within any given scene” (Abstract) — is not supported by the formalization in Sec. 3.1. There, the scene state is defined solely by object poses and containment states, and atomic actions are parameterized object relocations; the assertion that this interaction cycle “inherently defines the space of all possible tasks” (Sec. 3.1 and Fig. 2) is a definitional choice rather than a derived result, and no argument is given that every feasible mobile manipulation task reduces to object relocation. Feasible tasks such as turning on a lamp, opening a drawer, or pouring from a pitcher are not expressible in this formalism, and Sec. 6 confirms that non-pose state changes and fine-grained manipulation are future work. Because near-exhaustiveness is the paper's primary novelty and scaling argument, the abstract and introduction should be re-scoped to “near-exhaustive within the class of tasks achievable by navigate/pick/place primitives,” with that caveat carried consistently into the conclusion.","section":"Sec. 1; Sec. 3.1; Sec. 6"},{"comment":"The human-verified validity rates in Table 2 are the principal evidence for the feasibility of the generated tasks, but they are reported as bare point estimates: no sample size, no confidence interval, no annotator agreement, and no description of how tasks or annotators were selected for verification. With an unknown N, a rate of 94.0% could range from 47/50 to 940/1000, and the difference from the 29.4% baseline cannot be assessed for statistical significance; without agreement measures, the objectivity of the labeling is unverifiable. Please report per-condition sample sizes, the number of annotators and their agreement (e.g., Cohen's kappa), the sampling procedure (e.g., random stratified by level), and confidence intervals around the validity rates.","section":"Sec. 4, Table 2"}],"minor_comments":[{"comment":"There is a duplicated word: “explicitly encodes the the properties of every object” should read “the properties.”","section":"Sec. 3.3"},{"comment":"“The benchmark is constructed within an simulators [43]” is ungrammatical; it should be “within a simulator” or “within simulators.”","section":"Sec. 5.1"},{"comment":"The diversity comparison is based on 100 sampled tasks and presented as qualitative “lightmap” visualizations; a quantitative diversity metric (e.g., number of unique objects and receptacles covered per 100 tasks, with the sampling procedure and variability across samples) would make the diversity claim testable.","section":"Sec. 4, Fig. 4"},{"comment":"Level 4 (outcome-based) tasks are excluded from the benchmark (“We leave benchmarking Level 4 tasks in future work”), yet the abstract says the framework “thoroughly evaluat[es]” agents; the abstract and introduction should state that the reported evaluation covers only the process-based levels.","section":"Sec. 5.1.2"},{"comment":"The improvement results are based on 10 trial episodes and 100 Level-1 test tasks; please state whether the numbers are single runs or averages over seeds, and add variance or per-seed results so the improvements can be interpreted.","section":"Sec. 5.2.2, Table 4"},{"comment":"Comparing total task counts across datasets with very different numbers of scenes (e.g., M3Bench with 119 scenes versus 3 scenes here) conflates scale with density, and the “+∞” entry for ManiTaskGen-RAS is not a meaningful comparison to finite dataset sizes.","section":"Table 1"},{"comment":"The failed-episode transcript is internally inconsistent with the successful-episode transcript: the task description shown at “Steps used: 1/20” is “Move kitchenware_white_paper_towel_holder…” although the episode's stated task is “Move Remote to Bowl's rear-left receptacles,” and the action names differ across the two transcripts (show_receptacle_of_object vs. show_receptacles_of_object; change_view vs. rotate_observation_view_of_current_platform); these transcripts appear to be taken from different runs and should be made consistent.","section":"Appendix D.2"},{"comment":"The outcome-based task filter uses an ensemble of GPT-4o, Gemini-2.5-pro, and Claude-3.7-sonnet — the same model families later benchmarked in Sec. 5.1; the impact is limited because the benchmark covers only Levels 1–3, but the overlap should be explicitly acknowledged, and the filter's acceptance rate (how many template instantiations were rejected) should be reported.","section":"Sec. 4, Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"The core system is a genuine contribution, but the headline “near-exhaustive” claim needs re-scoping before the paper can be accepted; the fix is localized and I expect a major revision to succeed. I would also encourage the authors to report the statistical details of the human verification, since referees of the final version may otherwise discount the validity evidence. The manuscript is well suited to a robotics or embodied-AI venue given its systems-oriented evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper builds a real, working system for generating pick-and-place task instructions from scene graphs, and the 40K-task dataset plus benchmark harness are genuine assets. The weak spot is the headline claim of near-exhaustive task coverage, which is stated more broadly than the formalism supports. Section 3.1 explicitly scopes to agents with navigate, pick, and place skills, and the limitations section concedes that non-pose changes and fine-grained manipulation are future work. But the abstract and introduction promise \"the full spectrum of feasible tasks\" and a \"universal task generation framework\" without that caveat. That mismatch is fixable, not fatal.\n\nWhat's actually new: the Receptacle-Aware 3D Scene Graph with directional receptacle segmentation, the four spatial strategies for process-task generation, the 200 human-curated outcome templates filtered by VLM voting, and the clean process/outcome taxonomy. The human validity results beat the GPT baseline by a wide margin (94% vs 29.4% for process tasks, 86.5% vs 21.1% for outcome tasks), which is real evidence that the generation mechanism works. The benchmark harness with four difficulty levels, automatic success verification, and the failure analysis showing placement as the main bottleneck is also solid.\n\nSoft spots, in proportion: Table 2 reports no sample sizes, confidence intervals, or annotator agreement, so the validity numbers are hard to fully trust. The dataset uses only one scene per environment, and the SUN-RGBD scene had overlapping objects removed during preprocessing, which could inflate validity. Level 4 outcome tasks are never benchmarked on agents, so the \"comprehensive evaluation\" only covers process-based levels. The improvement study uses 10 trial episodes, no control condition, and only Level 1 tasks; the gains could partly reflect prompt effects or regression to the mean. The outcome-task filter uses the same VLM families that are later evaluated as agents, which is a mild circularity. And no code or data is released yet, only a promise.\n\nOn the central claim: even within pick-and-place, \"near-exhaustive\" is argued from the interaction cycle but not proven. The system samples and chains feasible actions; it does not enumerate all possible sequences or predicates. So the claim should be softened to \"comprehensive for the defined skill set\" unless the authors add a completeness argument.\n\nWho this is for: people building benchmarks for embodied VLM agents, and anyone needing automated task generation for rearrangement. It deserves a serious referee. I'd send it to review with a request for code/data, tighter statistics, a re-scoped abstract, and ideally a control group for the improvement experiment.","headline":"A genuinely useful pick-and-place task generator with a solid 40K dataset, but the near-exhaustiveness claim overreaches beyond the formalism; still deserves review.","tokens_in":57825,"tokens_out":2532,"would_cite":true,"duration_ms":31252,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ManiTaskGen claims the full space of feasible mobile-manipulation tasks in a scene can be generated automatically from a receptacle-aware scene graph, yielding near-exhaustive benchmarks and training tasks.","keywords":["mobile manipulation task generation","receptacle-aware scene graph","vision-language models","embodied decision-making","automatic benchmark construction","pick-and-place tasks","outcome-based task templates"],"falsifier":"Take a scene with a closed drawer, an unlit lamp, and a loaf of bread with a knife; generate ManiTaskGen's full task set for it and check every instruction. If no generated instruction requires a non-pose state change, such as switching on, opening, pouring, or cutting, then the task set is not near-exhaustive over what a capable embodied agent could do in that scene.","tokens_in":56662,"feed_emoji":"🤖","tokens_out":7252,"duration_ms":70873,"temperature":0.7,"pith_summary":"This paper introduces ManiTaskGen, a generator that takes object poses, bounding boxes, and optionally mesh models, and automatically produces a large, diverse set of feasible mobile manipulation tasks for any scene. It argues that the agent-environment interaction cycle, where each atomic object relocation updates the scene state and determines the next feasible actions, defines the space of all possible tasks for an agent with navigate, grasp, and place skills. From three scenes the system curates 39,871 tasks, with human-verified validity rates of 94.0% for process-based instructions and 86.5% for outcome-based instructions, well above a GPT-based baseline. The same generated tasks are used to build an automatic benchmark, where current vision-language agents succeed on fewer than 30% of episodes, and an inference-time self-reflection loop that raises success rates by 10 to 22 percentage points. If the claims hold, scene diversity no longer has to be capped by manual task annotation, and any new scene can become a source of evaluation and improvement data.","feed_headline":"One room becomes 40,000 robot tasks on demand","feed_subtitle":"Near-exhaustive pick-and-place task sets let VLM agents be benchmarked and improved with no hand-written instructions.","key_machinery":"The load-bearing object is the Receptacle-Aware 3D Scene Graph, a containment tree of scene objects whose free surface areas are segmented into anchor-relative directional receptacle regions, with empty surfaces divided into a 3x3 grid and adjacent regions mergeable into larger ones. It carries the argument by making the set of feasible atomic actions enumerable: every object-receptacle pair with sufficient space yields a candidate relocation, and simulating those relocations produces the state transitions needed to chain multi-step process tasks. The same graph supplies the platforms and objects that instantiate outcome-based task templates, while a 200-template curated dataset combined with VLM voting provides the abstract instruction coverage.","core_discovery":"The central claim is that the task space for an embodied mobile manipulation agent with navigate, grasp, and place primitives is captured by the discrete interaction cycle: at each step the scene state, defined by object poses and containment states, restricts the set of feasible atomic actions, defined as parameterized object relocations. All tasks are then either feasible sequences of such actions, called process-based tasks, or reachable target states, called outcome-based tasks. ManiTaskGen operationalizes this by constructing a Receptacle-Aware 3D Scene Graph that segments free surface space into directional receptacle regions, enumerating every feasible object-to-receptacle relocation, and chaining these relocations through simulated state updates to generate multi-step instructions. Outcome-based tasks come from instantiating a set of 200 human-derived templates and filtering with a multi-model VLM vote. The authors report automatic generation of 39,871 tasks from three scenes, validity rates of 94.0% and 86.5%, low average success for current VLMs on the resulting benchmark, and measurable gains from self-reflection using the generated tasks.","pith_inferences":["Editorial extension: the interaction-cycle argument generalizes; if atomic actions were extended to articulation and state toggles, the same graph-based enumeration would define a larger task space that remains near-exhaustive by the paper's own logic.","Editorial extension: the automatic success predicates on process-based tasks turn the generator into a reward source, so generated tasks could support supervised or reinforcement fine-tuning of policies, not just inference-time reflection.","Editorial extension: because Level 4 outcome tasks are not yet benchmarked, a human-judgment or learned-reward protocol is the natural next step, and the paper's template dataset makes that comparison reproducible.","Editorial extension: a completeness measure could be defined as the fraction of reachable distinct scene states represented by generated tasks, allowing near-exhaustiveness to be quantified beyond the 40K sample."],"forward_implications":["Any new scene with object poses, bounding boxes, and optionally meshes can be converted into a large benchmark without manual annotation, so task diversity no longer scales with human effort.","Process-based tasks come with automatically checkable success conditions, enabling both final and intermediate-step scoring for embodied agents.","Current vision-language agents succeed on under 30% of sampled generated tasks, with wrong placement the dominant failure, giving a concrete target for spatial-reasoning progress.","An inference-time self-reflection loop that uses generated tasks and their automatic evaluations improved success rates by 10 to 22 percentage points and transferred from one scene to another.","Outcome-based tasks add abstract instruction diversity such as clearing or organizing a surface, although automatic evaluation of those abstract tasks is left to future work."],"supporting_citations":[{"why":"Supplies the simulated home scene whose object meshes let ManiTaskGen extract interior receptacle surfaces for multi-level placement tasks.","marker":"[23]"},{"why":"Supplies a second simulated scene used to confirm scene-agnostic generation and to test whether reflection trials transfer to a different scene.","marker":"[24]"},{"why":"Supplies real-world RGB-D scenes with pose and bounding-box inputs, supporting the claim that the generator works outside simulation.","marker":"[25]"},{"why":"The closest prior task-generation work, restricted to trajectory expansion in specific scenes, against which ManiTaskGen positions its full process- and outcome-based coverage.","marker":"[34]"},{"why":"The verbal self-reflection method that the inference-time improvement loop adapts to generated task evaluations.","marker":"[26]"},{"why":"The interleaved reasoning-and-acting pattern used alongside reflection in the agent-improvement pipeline.","marker":"[27]"},{"why":"The model behind the GPTTaskGen baseline for validity comparison and one member of the VLM voting ensemble that filters outcome-based tasks.","marker":"[40]"},{"why":"The simulator that hosts the automatically constructed benchmark and provides the agent-environment interaction loop.","marker":"[43]"}],"fun_headline_variants":["Auto task generator yields 40k robot tasks per scene","One scene, 40k robot tasks on demand","Task generator turns any scene into a training ground","Auto-generated tasks boost VLM agents' decisions","40k embodied tasks from just three scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Near-exhaustiveness holds only if every feasible task can be expressed as relocating objects between receptacle regions; non-pose changes like toggling, opening, pouring, or cutting are outside the generated space, as the paper's own limitations state.","fun_headline_variants_meta":{"raw":{"variants":["Auto task generator yields 40k robot tasks per scene","One scene, 40k robot tasks on demand","Task generator turns any scene into a training ground","Auto-generated tasks boost VLM agents' decisions","40k embodied tasks from just three scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1357,"prompt_tokens":1015,"completion_tokens":342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":631,"tokens_out":342,"duration_ms":3999,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:48:08.444865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a scene with a closed drawer, an unlit lamp, and a loaf of bread with a knife; generate ManiTaskGen's full task set for it and check every instruction. If no generated instruction requires a non-pose state change, such as switching on, opening, pouring, or cutting, then the task set is not near-exhaustive over what a capable embodied agent could do in that scene.","supporting_citations":[],"review_version":1}