{"id":"62c953ff-bb2a-4e05-8124-bed89175147f","arxiv_id":"2506.19498","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A zero-training framework that adaptively selects spatial representation extractors per object and per task stage improves real-world robot manipulation success and efficiency over fixed-representation baselines.","lead":"T-Rex lets a vision-language model choose, for each object and task stage, how to extract spatial information: a point, a vector, or a full 6D pose. On 15 real-world tabletop tasks it beat two fixed-representation baselines in success rate (60.7% versus 36.4% and 30%) with a slightly lower average completion time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are never shown to come from adaptive selection, because ablations change both the selection policy and the set of available tools; a fixed-toolkit selection-policy control is missing.","rationale":"The paper is honest about several limitations and the real-world evaluation is substantial, so I read the central claim in the strongest form the authors intend: the adaptive choice of spatial representation types and granularity is what produces the advantage. For that attribution to hold, the experiments must show that varying the selection policy while holding the available tools fixed changes performance. Table 2 does not do this: each 'w/o Toolkit' row removes the whole toolkit, so adaptivity and tool availability are perfectly confounded. A reviewer therefore cannot tell whether T-Rex wins because it selects the right tool for each task or simply because it has better tools (6D pose, local zoom) than the point/vector-only baselines. The formal selection criterion (Sec. 3.2.1) would have made the mechanism explicit, but it is not implemented: A.4 replaces it with a prompt to choose the simplest tool. This is not an internal contradiction that falsifies the results, but it does mean the causal claim 'task complexity determines representation type/granularity' has no direct quantitative support. The proposed concrete test is feasible because the toolkit and task suite already exist; it would settle the attribution cleanly. I therefore keep the reader's CONDITIONAL verdict, with the added condition that the authors provide a fixed-toolkit selection-policy ablation and report the actual selection distribution. This is a partial agreement with the reader's weakest assumption: the reader focused on dependence on a proprietary VLM and missing prompts, while I find the more specific evidential gap to be the absence of any control for adaptivity itself.","tokens_in":24287,"tokens_out":8348,"duration_ms":88727,"concrete_test":"Run a new ablation on the same 15 tasks x 10 trials with the Toolkit held constant and only the selection policy varied: (1) T-Rex's CoG-based adaptive selection; (2) a fixed 'most-capable' policy that always invokes the highest-capability applicable tool for each object (e.g., SE3PoseEstimator where a mesh exists, otherwise SemanticKeypointExtractor plus LocalSubImageExtractor when fine detail is required); (3) an oracle policy that chooses the per-task best tool using the authors' judgment; (4) random selection. Compare success rate and average completion time across the four policies. If adaptive selection does not beat both most-capable and oracle on success at comparable or lower time, the central attribution of the gain to adaptivity is unsupported. Also report the tool-selection distribution from CoG logs and check whether selection follows the Sec.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes T-Rex's 60.7% success and 45.5s average time to task-adaptive selection of representation types and granularities. But the experiments never isolate the selection mechanism from the contents of the Toolkit. In Table 2, 'w/o Toolkit (SP)' and 'w/o Toolkit (VPV)' remove the entire Toolkit, replacing it with one fixed extractor; this simultaneously deletes adaptivity and deletes the richer tools (6D pose, local sub-image/zoomed extraction, vectors, state/topological extractors). The 5.7-point gap between Ours (60.7%) and w/o Toolkit (VPV) (55.0%) could therefore be explained by the mere addition of SE3 pose and multi-granularity tools, not by selecting among them per task. No condition holds the toolkit fixed and varies only the selection policy (e.g., fixed-most-capable, random, or oracle), so the paper's strongest claim is underdetermined. This is compounded by the formal criterion in Sec. 3.2.1, argmax over P_succ - lambda*h_t, which is never operationalized: Appendix A.4 states the VLM is prompted to choose 'the simplest tool' whose representation satisfies the task, with no measured P_succ or specified lambda. The promised multi-granularity ablation referenced in Sec. 3.2.2 is also absent from Table 2.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes T-Rex, a framework for real-world robotic manipulation in which a Vision-Language Model (GPT-4.1) grounds a natural-language instruction through a four-stage Chain of Grounding (CoG), selects per-object, per-stage spatial representation extractors from an extensible toolkit (keypoints, vectors, 6D pose, local sub-image, state machines, topological orderings), and generates Python constraint functions that a low-level solver converts into end-effector trajectories. The main empirical claim is that task-adaptive selection of representation types and granularities yields better spatial understanding, efficiency, and stability than fixed-representation baselines VoxPoser and ReKep, with 60.7% success over 150 real-world trials in 15 tasks versus 36.4% and 30%, and lower average completion time (45.5 s versus 53.3 s and 50.8 s), all without additional training.","tokens_in":24533,"tokens_out":6085,"duration_ms":54352,"significance":"If the central claim is established, T-Rex would be a useful contribution: it is training-free, modular, and the reported aggregate gains over two fixed-representation baselines are large and consistent across several task categories. The paper also provides a useful decomposition of error sources (Fig. 6), a VLM-choice study (Table 3), and an honest discussion of limitations and dependencies on proprietary models. However, the experiments as reported do not yet isolate the adaptive selection mechanism from the contents of the toolkit, the formal selection criterion in Sec. 3.2.1 is not operationalized, the efficiency numbers in Table 1 are internally inconsistent, and the claimed multi-granularity ablation is missing. These issues leave the paper's strongest claim underdetermined rather than refuted.","major_comments":[{"comment":"The central claim that task-adaptive selection drives the improvement is not isolated by the ablations. The 'w/o Toolkit (SP)' and 'w/o Toolkit (VPV)' conditions remove the entire toolkit and replace it with one fixed extractor, so they change both the selection policy and the set of available tools; the 5.7-point gap between Ours and w/o Toolkit (VPV) could be due to the added SE3 pose and multi-granularity extractors rather than to selecting among them per task. Please add a control that holds the toolkit fixed and varies only the selection policy (e.g., always use the most capable tool, or a random or oracle selector), or soften the causal claim accordingly.","section":"Sec. 4.2, Table 2"},{"comment":"The formal tool-selection criterion t* = argmax [P_succ - lambda*h_t] is not operationalized anywhere. P_succ is never estimated, lambda is never set, and Appendix A.4 instead states that the VLM is prompted to choose 'the simplest tool' whose representation satisfies the task. This is not merely a presentation gap: the paper's efficiency-accuracy trade-off claim depends on that criterion. Please either specify how P_succ and lambda are computed or calibrated, or revise the formalization to describe the heuristic actually used.","section":"Sec. 3.2.1, Eq. (1); Appendix A.4"},{"comment":"The text in Sec. 3.2.2 says that fine-grained extraction from the raw view 'frequently fail[s]... as evidenced by our ablation studies in Sec. 4.2,' but Table 2 contains no row that toggles the multi-granularity (local sub-image) mechanism while keeping the rest of the pipeline fixed. The promised ablation is absent, so the necessity of multi-granularity extraction is not supported by the reported data. Please add the ablation or remove the reference.","section":"Sec. 3.2.2; Sec. 4.2"},{"comment":"The reported total average completion times do not match the arithmetic means of the per-task times. For T-Rex the per-task times sum to 637.4 s across 15 tasks, giving a mean of 42.5 s, not 45.5 s; analogous discrepancies appear for VoxPoser (47.4 vs 50.8) and ReKep (49.7 vs 53.3). Since efficiency is one of the three headline claims, please clarify how the total times are computed (e.g., including setup time, conditioning on success, or using different trial sets) and correct any inconsistency.","section":"Table 1"},{"comment":"All per-task results are based on 10 trials, and no significance testing or confidence intervals are reported for either success rates or completion times. With n = 10, several per-task comparisons are within the range of sampling noise (e.g., Stack Block 6/10 vs 6/10; Sweep Trash 9/10 vs 8/10), and the claim of 'significant advantages' needs at least exact binomial or bootstrap intervals. Adding this analysis does not require new experiments, but the statistical support should match the strength of the wording.","section":"Sec. 4.1, Table 1"}],"minor_comments":[{"comment":"The conclusion says 'Extensive experiments in both simulated and real-world robotic environments,' but the paper reports only real-world experiments. Please correct the statement or add simulation results.","section":"Sec. 5 / Conclusion"},{"comment":"There are typographical errors such as 'Rpresentation' in the Sec. 3.2 heading and 'V oxPoser' with stray spaces in several places. Please proofread.","section":"Sec. 3.2 heading; throughout"},{"comment":"The method SPAR3D is written 'SRAR3D' in one sentence. Please unify the spelling.","section":"Appendix A.5.2"},{"comment":"The registry entries are not valid JSON: several entries are missing commas or closing braces (e.g., the Local SubImage Extractor and VLM Task Vector Extractor entries). If the registry is meant to be machine-readable, provide a validated example.","section":"Appendix A.5.1, Listing 3"},{"comment":"Stray annotations such as 'zi too small' and 'setup 1 duan' appear in or near Figure 6 and the experimental text. These should be removed or fixed before submission.","section":"Fig. 6 and main text"},{"comment":"The 'Inference Validity Rate' is human-evaluated, but the evaluation protocol, the number of annotators, and the exact rubric are not described. Please add details.","section":"Table 3"},{"comment":"There is a typo: 'seletions' should be 'selections'. Also, the notation for the CoG mapping is introduced twice (Sec. 3.3 and A.9) with slightly different formatting; please standardize.","section":"Appendix A.9"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and represents a useful systems contribution, but the headline causal claim ('task-adaptive selection drives the gains') is not yet supported by the experimental design; the missing fixed-toolkit selection-policy control is the single most important addition. The Table 1 time inconsistency is a data-integrity issue that must be resolved. There is also a mismatch between the paper's claim of an 'open' Spatial Representation Extraction Toolkit and the absence of a code or model release link; the authors should state availability explicitly. I do not see grounds for rejection, but the requested additions are substantial enough to warrant a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick read. T-Rex is a clean, honest system paper. What's actually new is letting a VLM pick, per object and stage, from a registry of spatial extraction tools (center point, semantic keypoints, 6D pose, sub-image crops, VLM-generated points/vectors, state machines, topo orderings), with a staged Chain-of-Grounding prompt to keep that selection reliable. The experiments are real: 15 manipulation tasks, 10 trials each, randomized object poses, and a decent aggregate win—60.7% versus 36.4% for ReKep(Auto) and 30% for VoxPoser, with lower average time. I see no reason to doubt the numbers; baselines are reasonable.\n\nGive credit where due. The ablations show CoG helps, the toolkit helps, the error breakdown is informative, and the appendix is candid about tracking being the bottleneck and about borrowing the solver from ReKep. That's good practice.\n\nThe soft spot is load-bearing, and the stress-test note is right. The paper's title claim is task-adaptive selection, but the ablations never vary selection policy with the toolkit held fixed. 'w/o Toolkit (VPV)' removes the whole toolkit, so the 5.7-point gap to Ours could come from having SE3 pose and sub-image tools available at all, not from choosing among them per task. You need a fixed-toolkit control (always strongest, or random) to isolate adaptivity. Without it, the 60.7 vs 55.0 comparison doesn't support the central claim.\n\nSmaller issues: the formal argmax criterion in Sec. 3.2.1 is never operationalized—Appendix A.4 has the VLM pick 'the simplest tool,' a different criterion. The promised multi-granularity ablation doesn't appear in Table 2. Ten trials per task with no significance testing is thin for per-task claims. No code, data, or exact prompts are released. The conclusion mentions simulated experiments, but the paper contains none. And the 6D pose tool needs per-object meshes, which cuts against open-vocabulary framing for pose-heavy tasks.\n\nNet: a useful engineering contribution that deserves a serious referee, but the headline claim needs either reworking or more evidence. For people building zero-shot VLM manipulation pipelines, the toolkit-plus-CoG recipe is immediately usable; for the scientific claim about adaptivity, treat it as a hypothesis. I'd send it to review, asking for a fixed-toolkit selection control, released prompts/code, and the missing granularity ablation. I wouldn't cite the 60.7% number as evidence for adaptivity until that control exists.","headline":"Useful system paper with a real robot gain, but the paper's headline claim—that adaptive selection, not just a bigger toolkit, drives the gain—is not actually tested.","tokens_in":25124,"tokens_out":4102,"would_cite":true,"duration_ms":42253,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"T-Rex claims that letting a vision-language model choose a spatial representation per task, per object, and per stage—points for simple placement, vectors for orientations, 6D poses for fine alignment—gives a manipulation system better…","keywords":["task-adaptive spatial representation extraction","vision-language models for robotics","robotic manipulation","Chain of Grounding","6D pose estimation","keypoint extraction","zero-shot robot control","spatial constraint generation"],"falsifier":"Run the same 15 tasks on the same robot with the same toolkit and the same Chain of Grounding prompts, but replace the adaptive tool selection with each single fixed extractor (points only, vectors only, 6D poses only) and check whether adaptive selection still beats the best fixed configuration. Since the paper's own ablation shows a fixed simplest point extractor drops success to 30.7 percent and a fixed point-and-vector extractor to 55.0 percent, a narrower observed gap would indicate the advantage comes from the toolkit's stronger extractors rather than from adaptation.","tokens_in":24055,"feed_emoji":"🤖","tokens_out":9350,"duration_ms":88666,"temperature":0.7,"pith_summary":"The paper argues that a robot manipulation system guided by a vision-language model should not commit to one spatial representation—points, vectors, or 6D poses—for every task. Instead, it should pick the type and granularity of representation per object and per task stage, because simple tasks only need a center point while orientation-sensitive tasks need a full 6D pose, and richer representations cost more time. The paper's system, T-Rex, operationalizes this by letting the VLM choose tools from an extensible registry under a step-by-step Chain of Grounding prompt, then generating Python constraint functions and a low-level action sequence. In 150 real-world trials across 15 everyday tasks, it reports 60.7% success versus 36.4% for a fixed keypoint-constraint baseline and 30% for a value-map baseline, with a lower average completion time. The central claim is that the adaptive selection mechanism itself, not any single extractor, delivers the gain, and all of this happens without additional training.","feed_headline":"Per-task spatial extractors lift robot manipulation to 60.7 percent","feed_subtitle":"Choosing among points, vectors, and 6D poses per subtask beats fixed extractors in 150 real-world trials.","key_machinery":"The load-bearing mechanism is the pairing of two components: an extensible Spatial Representation Extraction Toolkit—a registry of extractors (center points, semantic keypoints, uniform scene keypoints, 6D pose, local subimage crops, and VLM-generated point, vector, state-machine, and topology tools), each annotated with inputs, output format, and historical average execution time—and the Chain of Grounding (CoG) prompt structure that takes a natural-language instruction and a scene image and forces the VLM through four phases: operational hints, natural-language constraints, per-object tool selection, and Python constraint-code generation. The formal selection criterion is $t_{s,o}^{*}=\\arg\\max_{t\\in\\mathcal{R}}\\left[P_{\\mathrm{succ}}(t\\mid I,X,s,o)-\\lambda h_{t}\\right]$, balancing estimated extraction success against average running time; the paper notes that in practice the VLM is prompted to choose the simplest tool whose extracted representation satisfies the task requirements. This mechanism produces heterogeneous, multi-granular spatial representations that are then consumed by a solver or by generated policy code to yield end-effector trajectories.","core_discovery":"T-Rex's central discovery is that task complexity determines the type and granularity of spatial representation needed, and that stronger representational power carries higher extraction cost; therefore choosing the simplest sufficient extractor for each entity at each stage yields both better spatial understanding and better efficiency. Concretely, the framework maintains an extensible toolkit registry of extractors, and the Chain of Grounding prompt makes the VLM decompose the instruction into stages, infer operational hints, select a tool for every task-relevant object (preferring the simplest that satisfies the requirement), and emit per-stage Python constraint functions. The resulting heterogeneous, multi-granular representations feed a solver or a generated policy script to produce end-effector trajectories. The paper reports 60.7% average success over 150 trials (15 tasks times 10 trials) compared with 36.4% for the automatic keypoint-constraint baseline and 30% for the value-map baseline, while taking 45.5 seconds on average versus 53.3 and 50.8 seconds. It concludes that the adaptive selection mechanism—supported by Chain of Grounding—is the source of the improvement, not any individual extractor.","pith_inferences":["Beyond the paper: the same selection logic could be ported to other VLM-driven constraint-manipulation pipelines, which suggests the transferable contribution is the selection policy itself, not the particular extractors in the toolkit.","Beyond the paper: because the formal argmax criterion is approximated by a 'choose the simplest tool' prompt, a learned success-probability estimator trained on logged tool executions could make the method dependable with weaker or cheaper VLMs.","Beyond the paper: the ablation spread—30.7 percent with a fixed point extractor, 55.0 percent with a fixed point-and-vector extractor, 60.7 percent with adaptation—predicts that adaptation matters most on tasks mixing coarse positioning with fine orientation requirements, and least on purely positional tasks.","Beyond the paper: because the paper reports success varying from 55.0 to 61.4 percent across different vision-language models, it implies that the current ceiling is set by model inference quality; stronger spatial reasoning in future VLMs should raise T-Rex's ceiling without changing its architecture."],"forward_implications":["New extractors, including tactile or future pose-estimation modules, can be added to the registry and become available to the planner without retraining the system.","Tasks that mix coarse positioning with fine orientation requirements, such as placing plush toys upright and aligned, can use cheap point representations for positioning and 6D poses only where orientation matters, avoiding the time cost of full-pose extraction on every object.","Because the toolkit chooses the simplest sufficient tool, the paper's average completion time over the 15 tasks falls to 45.5 seconds, about 8 seconds faster than the keypoint-constraint baseline and 5 seconds faster than the value-map baseline.","Removing Chain of Grounding drops success to 52.1%, so the explicit step-by-step grounding of instructions into hints, constraints, tool choices, and code is load-bearing for the method's stability.","The system claims plug-and-play deployment with no additional training, meaning its performance ceiling is set by the reasoning quality of the vision-language model and the quality of registered extractors, not by robot-specific data collection."],"supporting_citations":[{"why":"Supplies the keypoint-constraint baseline whose fixed extraction scheme T-Rex outperforms, and the solver-based low-level action generation design.","marker":"[2]"},{"why":"Supplies the value-map baseline used as the second fixed-representation comparison target.","marker":"[14]"},{"why":"Supplies the code-as-policies route used when constraints depend on non-conventional representations.","marker":"[44]"},{"why":"Grounding DINO is the open-set detector behind center-point and local-subimage extraction in the toolkit.","marker":"[52]"},{"why":"SAM6D is the 6D pose estimator whose inclusion lets T-Rex succeed on orientation-demanding tasks.","marker":"[60]"},{"why":"DINOv2 features are combined with SAM and K-means in the semantic keypoint extractor.","marker":"[71]"},{"why":"Provides the segmentation model used for semantic keypoints and structure in the toolkit.","marker":"[55]"},{"why":"Set-of-Mark prompting is the basis of the uniform scene keypoint tool.","marker":"[72]"}],"fun_headline_variants":["Adaptive spatial extractors boost robot task success to 60.7%","Task-aware extractor selection lifts robot success to 60.7%","T-Rex chooses simplest spatial cue per task for faster, better robots","Robot manipulation: selecting spatial extractors per task hits 60.7%","Adaptive per-task spatial cues improve robot manipulation to 60.7%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole system succeeds only if a proprietary vision-language model, prompted by the Chain of Grounding routine, reliably decomposes instructions, chooses the right extractor from the registry, and writes correct Python constraint functions with no training—if that model or prompt fails, the claimed advantage collapses.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive spatial extractors boost robot task success to 60.7%","Task-aware extractor selection lifts robot success to 60.7%","T-Rex chooses simplest spatial cue per task for faster, better robots","Robot manipulation: selecting spatial extractors per task hits 60.7%","Adaptive per-task spatial cues improve robot manipulation to 60.7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2679,"prompt_tokens":998,"completion_tokens":1681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":1582}},"tokens_in":614,"tokens_out":1681,"duration_ms":10524,"temperature":1.0,"reasoning_tokens":1582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:38.854930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 15 tasks on the same robot with the same toolkit and the same Chain of Grounding prompts, but replace the adaptive tool selection with each single fixed extractor (points only, vectors only, 6D poses only) and check whether adaptive selection still beats the best fixed configuration. Since the paper's own ablation shows a fixed simplest point extractor drops success to 30.7 percent and a fixed point-and-vector extractor to 55.0 percent, a narrower observed gap would indicate the advantage comes from the toolkit's stronger extractors rather than from adaptation.","supporting_citations":[{"cited_title":"Sam-6d: Segment anything model meets zero-shot 6d object pose estimation","cited_arxiv_id":null,"evidence_quote":"SAM6D is the 6D pose estimator whose inclusion lets T-Rex succeed on orientation-demanding tasks."}],"review_version":1}