REVIEW 5 major objections 5 minor 33 references
This paper shows that unscripted internet photos of tabletops can be automatically converted into 100,000 collision-free physics-ready scenes for robot training.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 08:40 UTC pith:MCCYYVHZ
load-bearing objection TableVerse is a serious engineering contribution with a load-bearing evaluation gap: the collision-free guarantee is by construction, and the 'grounded' fidelity claim rests on an unverified SAM3D reconstruction stage that the paper itself concedes can fail. the 5 major comments →
TableVerse: A Large-scale Tabletop Dataset with Real-world Grounded Layouts for Generalizable Manipulation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that real-to-sim for tabletop scenes can be made fully automatic and deterministic: from a single unscripted image, the pipeline extracts instances with an open-vocabulary detector and a segmentation model, reconstructs each asset as a separate mesh — including decomposing containers into their nested contents, which are then assembled by a brief free-fall physics simulation — and recovers metric scales and 6-DoF poses from a monocular depth point cloud after gravity alignment. The key to making these raw reconstructions simulable is the Layout-Consistent Collision Rectification module, which groups touching objects, lands them on the table plane, expands them radially a
What carries the argument
The load-bearing component is the Layout-Consistent Collision Rectification (LCCR) module: a geometric optimization that resolves mesh interpenetrations while preserving the macroscopic spatial arrangement. It works in three phases — hierarchical contact grouping and grounding to the table plane; construction of a radial scene graph that expands each group outward from the most central cluster along its original azimuthal heading until it clears already-settled neighbors; and vertical lifting of the smaller-footprint object to clear remaining stacking intersections. A short forward physics simulation then closes micro-gaps so objects rest in stable contact. LCCR, together with the preceding
Load-bearing premise
The whole pipeline assumes the single-view mesh reconstruction produces 3D models that faithfully match the real objects; the paper's own limitation section states that objects inside containers, being low-resolution and few-pixel, can cause the reconstructor to generate completely different objects — so if that happens at scale, the dataset's claimed authentic topologies and metric fidelity silently degrade even while collision rates stay at zero.
What would settle it
3D-scan a set of real tabletops with a depth sensor to obtain ground-truth geometry, run the pipeline on the same single views, and measure per-instance mesh fidelity (e.g., Chamfer distance or volumetric IoU) between reconstructed and scanned assets. The pipeline's collision rejector can keep collisions at 0% even when shapes are wrong, so a strong divergence for low-resolution or occluded objects would refute the 'authentic topologies' part of the claim. A simpler check: sample a random subset of TableVerse-100K assets and ask humans whether each mesh recognizably matches the object in the s
If this is right
- Training data for manipulation policies can be scaled nearly without bound: any internet photo of a tabletop can be turned into an interactive scene, so the 100K corpus is a floor, not a ceiling.
- The 0.0% collision rate on in-the-wild images means generated scenes can be loaded directly into a physics engine, removing the manual cleanup that blocks earlier automated pipelines.
- Because the dataset pairs each scene with instruction-conditioned pick-and-place trajectories, it supports learning language-conditioned visuomotor policies, not just static scene understanding.
- The long-tail distribution — roughly one million instances across over 35,000 categories — should expose policies to the clutter and rarity patterns of real human environments rather than the tidy synthetic layouts used in prior benchmarks.
- Decomposing composite containers into independent parts makes contents 'tangible' in simulation, enabling manipulation tasks that involve objects inside containers, a category earlier datasets largely ignored.
Where Pith is reading between the lines
- Because the fidelity of each reconstructed mesh is only as good as the single-view reconstruction stage, the dataset's real-world 'authenticity' is likely to be strongest for large, unoccluded objects and weakest for small items inside containers — exactly the part of the distribution the paper itself flags as failure-prone.
- A policy trained on TableVerse-100K and evaluated in the physical world on the same objects would be the decisive test; the paper does not yet report that transfer, so the sim-to-real value remains an inference.
- The MLLM-based evaluation and annotation could itself be a source of bias: if the scoring model prefers visually tidy scenes, the curation may filter out some of the chaotic layouts the dataset claims to represent.
- If the pipeline is truly zero-human-intervention, the same architecture could be pointed at other structured domains (shelves, warehouses, kitchens) to produce analogous large-scale datasets, a generalization the authors do not claim but the method invites.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces TableVerse, a fully automatic Real2Sim pipeline that takes a single in-the-wild image of a tabletop and produces a MuJoCo simulation scene with object meshes, metric poses, and collision-free pick-and-place trajectories. The pipeline combines Seed-1.8 open-vocabulary detection, SAM2 masks, SAM3D mesh reconstruction, Depth Anything 3 point clouds, a Layout-Consistent Collision Rectification (LCCR) step, MuJoCo settling, and a Gemini 2.5 Pro curation/annotation stage. The authors use it to build the claimed TableVerse-100K dataset of 100,000 scenes, ~1M instances, and 35K categories, and report 0.0% Scene Collision Rate and superior GPT-Scores on a 100-scene test set relative to MIDI, SAM3D, and SceneMaker.
Significance. If the pipeline lived up to its claims, TableVerse would be a meaningful step toward scalable, real-world-grounded simulation data for manipulation research. The paper deserves credit for a complete, well-described pipeline with detailed prompts in the appendices, for processing unscripted internet imagery rather than only synthetic layouts, and for a deterministic geometric collision-rectification stage that could be useful on its own. However, the current evidence does not establish the central claims: the collision-free rate is a tautology, mesh fidelity is unverified and explicitly partial, the MLLM evaluation lacks blinding/calibration/human agreement, and no trajectory statistics or dataset breakdowns are given. These are not mere presentation issues; they are load-bearing for the paper's stated contributions.
major comments (5)
- [§3.2, Eq. (2); Tables 1 and 2] The headline 0.0% Scene Collision Rate is an algorithmic invariant, not an empirical result. Eq. (2) defines d* as the smallest nonnegative translation along the radial direction that makes the group disjoint from the anchor set, so the LCCR output is collision-free by construction. Reporting this as a comparative result against MIDI, SAM3D, and SceneMaker in Table 1 is misleading: those methods are evaluated on raw reconstructions without any analogous rectification step. The ablation in Table 2 (Direct 79% -> LCCR 0%) is simply a check that the optimization terminates. Please reframe this metric as a post-condition, and instead evaluate physical validity after simulation (e.g., fraction of scenes that remain stable, objects stay in place, no explosion) or report the collision rate before rectification for all methods.
- [§3.1 and §5] The dataset's core value claim is that reconstructed meshes are 'authentic topologies' with 'accurate metric scales.' This rests on SAM3D's single-view reconstruction fidelity, but §5 explicitly concedes that low-resolution objects inside containers 'occupy fewer pixels, causing SAM3D to generate completely different objects.' No quantitative fidelity evaluation is provided: no Chamfer distance against ground-truth scans, no human verification rate, no semantic correctness statistics. Since container nesting is a common tabletop configuration, this failure mode can affect a large fraction of the dataset. The GPT-Score cannot resolve this because it is a holistic MLLM judgment, not a geometric metric. Please add a fidelity evaluation on a labeled subset and report the failure-rate effect on downstream trajectory validity.
- [§4.1, Appendix C.2] The GPT-Score evaluation is vulnerable to ordering and identity bias. The prompt feeds a stitched 2x5 grid where the reference and the four methods are always in fixed columns, with TableVerse always at column 5. The MLLM is thus not blind to which output is which. Moreover, the same Gemini 2.5 Pro model family is used both for dataset curation and for evaluation. No calibration or human-agreement analysis is reported for the LF/VQ/GQ scores. Please randomize and blind the column order, repeat each evaluation multiple times to measure stability, and report human agreement on a subset. Without these controls, the quantitative superiority in Table 1 is not a reliable scientific claim.
- [§3.4 and §4.2] The paper claims to provide 100K scenes 'paired with continuous expert pick-and-place trajectories' and describes an automated trajectory generation pipeline, but no trajectory-level validation is reported. There is no planning success rate, no statistic on how many MLLM-generated task instructions were infeasible or produced failed cuRobo plans, and no collision check on the final trajectory corpus. Given that tasks are generated from partial, single-view renders and the target objects may be misreconstructed (see limitation), the trajectory corpus may contain systematic failures. Please report success rates and fallback distributions (Stages 1-3 in Appendix D.1) and provide some qualitative examples of trajectory execution.
- [§3.3 and Figure 1] The scale claims (100K environments, ~1M instances, 35K categories) are not supported by any quantitative breakdown. A dataset paper of this ambition should include a data card: number of images sourced, per-stage pass rates (detection, reconstruction, usability gate, stabilization), category frequency histogram, scene complexity distribution, and uniqueness checks. Without such statistics, the 'long-tail diversity' and 'physical consistency' statements cannot be evaluated. Please add this analysis or explicitly label the numbers as planned totals rather than verified counts.
minor comments (5)
- [Table 1 and throughout] 'GPT-Score' is produced by Gemini 2.5 Pro, not GPT; consider renaming to 'MLLM-Score' or 'Gemini-Score' for accuracy.
- [§3.1 vs Appendix A] Section 3.1 says non-meshable substances 'fall back to regular single entities,' which appears to conflict with Appendix A's instruction to exclude liquids and include only the container as a regular object when it contains liquids. Please align the wording.
- [Data availability] The paper states the dataset is released but only gives a project page; provide a direct download URL and a data-use license to support the open-science claim.
- [Figure 1] The statistical overview would be more informative with axis labels and exact numbers; consider adding subplots for category distribution and per-scene object count.
- [§4.1] 'Scene Collision Rate' is defined as clipping before physics relaxation; please state explicitly that this rates geometric disjointness, not physical stability.
Circularity Check
Collision-free result is tautological: LCCR Eq. (2) defines the displacement that makes scenes disjoint, so the reported 0.0% Scene Collision Rate is guaranteed by construction; fidelity claims are separately unvalidated but not circular.
specific steps
-
self definitional
[Section 3.2, Eq. (2); Section 4.1/4.2 (Scene Collision Rate metric and Table 1)]
"d∗(k) = min{d≥0|(G(k) +d⃗ v(k))∩ Ak−1 =∅}(2) ... Scene Collision Rate (%) ↓: The percentage of generated scenes exhibiting volumetric mesh interpenetration (clipping) prior to physics relaxation"
Eq. (2) defines the LCCR displacement as the smallest d for which the translated group is disjoint from all previously settled meshes; applying this translation makes the output collision-free by construction. The 'Scene Collision Rate' metric then reports whether this construction was applied, so Table 1's 0.0% value (and the 0.0% row in Table 2 for Direct + LCCR) is an algorithmic invariant rather than an empirical evaluation of reconstruction quality. Using this as a headline experimental result is equivalent to restating the optimization objective.
full rationale
The only circular reduction I can substantiate from the paper's own equations is the Scene Collision Rate. Eq. (2) explicitly searches for d* such that (G(k) + d v(k)) ∩ A_{k-1} = ∅, and every group is then translated by that amount, so any scene passing through LCCR is disjoint by construction. The ablation confirms this: Direct + LCCR already yields 0.0% before MuJoCo stabilization. Yet the paper presents 'an absolute 0.0% Scene Collision Rate' as a headline comparison in Table 1, which is circular as evidence for mechanical validity. I do not count the Gemini 2.5 Pro curation/evaluation loop as a formal circular step: using the same MLLM to annotate scenes and to score renderings is an evaluation confound, but there is no defined reduction of the GPT-Score to the curation labels. I also flag Section 5's admission that low-resolution container contents can cause SAM3D to 'generate completely different objects'; this undermines the 'authentic topologies' claim, but it is a fidelity/correctness risk rather than a circularity. No load-bearing self-citation chain is present: the external baselines and model cards are not used to define TableVerse's own results. Because the reconstruction, scale-alignment, and trajectory-synthesis components retain independent content, the circularity is partial, not total.
Axiom & Free-Parameter Ledger
free parameters (6)
- gamma_strict =
not specified
- epsilon =
not specified
- delta_margin =
not specified
- overlap_threshold =
50%
- quality_score_weights =
45/15/15/25
- label_confidence_threshold =
0.7
axioms (5)
- domain assumption Single-view monocular RGB plus Depth Anything 3 yields metric-scaled point clouds accurate enough for absolute object dimensions in dense clutter.
- domain assumption SAM3D reconstructs geometrically faithful meshes, including for occluded or low-resolution objects.
- domain assumption MuJoCo gravity settling after collision rectification yields mechanically stable states that faithfully represent real contact mechanics.
- domain assumption Gemini 2.5 Pro visual judgments are valid for scene usability, label correctness, physical properties, quality scoring, and task generation.
- ad hoc to paper Automatically filtered internet images are representative of real human tabletop environments.
read the original abstract
The development of generalizable robotic manipulation policies is inherently bounded by the availability of large-scale, high-fidelity scene data. While recent automated synthesis methods attempt to bridge this gap via text-to-layout hallucination or simplified procedural generation, they frequently suffer from physical implausibility and fail to capture the complex, dense clutter of actual human environments. In this paper, we introduce TableVerse, a fully automated Real2Sim pipeline that shifts the paradigm from imaginative layout generation to deterministic reconstruction from unstructured, in-the-wild image data. Our framework seamlessly processes unscripted internet media into high-fidelity, simulation-ready tabletop environments with accurate metric scales, authentic topologies, and verified mechanical stability. Furthermore, an automated task-conditioned trajectory generation framework is integrated to synthesize high-quality, collision-free pick-and-place demonstrations. Leveraging this complete pipeline, we construct the TableVerse-100K Dataset, a large-scale corpus comprising 100,000 unique, physically consistent environments paired with interactive manipulation trajectories. By capturing diverse asset compositions, realistic spatial distributions, and high-quality demonstrations, TableVerse-100K establishes a highly scalable and high-fidelity data foundation, providing significant value to facilitate future research in generalizable robotic manipulation tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Huang, Zehuan and Guo, Yuan-Chen and An, Xingqiao and Yang, Yunhan and Li, Yangguang and Zou, Zi-Xin and Liang, Ding and Liu, Xihui and Cao, Yan-Pei and Sheng, Lu , booktitle =
-
[2]
arXiv preprint arXiv:2511.16624 , year=
Sam 3d: 3dfy anything in images , author=. arXiv preprint arXiv:2511.16624 , year=
-
[3]
arXiv preprint arXiv:2512.01204 , year=
TabletopGen: Instance-Level Interactive 3D Tabletop Scene Generation from Text or Single Image , author=. arXiv preprint arXiv:2512.01204 , year=
-
[4]
arXiv preprint arXiv:2401.14159 , year=
Grounded sam: Assembling open-world models for diverse visual tasks , author=. arXiv preprint arXiv:2401.14159 , year=
-
[5]
arXiv preprint arXiv:2511.10647 , year=
Depth anything 3: Recovering the visual space from any views , author=. arXiv preprint arXiv:2511.10647 , year=
-
[6]
International Conference on Learning Representations , volume=
Sam 2: Segment anything in images and videos , author=. International Conference on Learning Representations , volume=
-
[7]
ACM Transactions on Graphics (TOG) , volume=
Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search , author=. ACM Transactions on Graphics (TOG) , volume=. 2022 , publisher=
2022
-
[8]
arXiv preprint arXiv:2512.10957 , year=
SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model , author=. arXiv preprint arXiv:2512.10957 , year=
-
[9]
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=
Holodeck: Language Guided Generation of 3D Embodied AI Environments , author=. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=
2024
-
[10]
ArXiv , year=
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models , author=. ArXiv , year=
-
[11]
Advances in neural information processing systems , volume=
Mesatask: Towards task-driven tabletop scene generation via 3d spatial reasoning , author=. Advances in neural information processing systems , volume=
-
[12]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Metascenes: Towards automated replica creation for real-world 3d scans , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[13]
2021 IEEE/CVF International Conference on Computer Vision (ICCV) , year=
3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics , author=. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , year=
2021
-
[14]
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year=
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , author=. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year=
2017
-
[15]
ArXiv , year=
IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation , author=. ArXiv , year=
-
[16]
ArXiv , year=
TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes , author=. ArXiv , year=
-
[17]
ArXiv , year=
ShapeNet: An Information-Rich 3D Model Repository , author=. ArXiv , year=
-
[18]
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year=
3D ShapeNets: A deep representation for volumetric shapes , author=. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , year=
2015
-
[19]
2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , year=
MuJoCo: A physics engine for model-based control , author=. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , year=
2012
-
[20]
2025 , eprint =
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author =. 2025 , eprint =
2025
-
[21]
2024 , howpublished =
2024
-
[22]
2026 , url=
Seed1.8 Model Card: Towards Generalized Real-World Agency , author=. 2026 , url=
2026
-
[23]
arXiv preprint arXiv:1703.09312 , year=
Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics , author=. arXiv preprint arXiv:1703.09312 , year=
-
[24]
The International Journal of Robotics Research , volume=
Grasp pose detection in point clouds , author=. The International Journal of Robotics Research , volume=. 2017 , publisher=
2017
-
[25]
2019 international conference on robotics and automation (ICRA) , pages=
Pointnetgpd: Detecting grasp configurations from point sets , author=. 2019 international conference on robotics and automation (ICRA) , pages=. 2019 , organization=
2019
-
[26]
2021 IEEE international conference on robotics and automation (ICRA) , pages=
Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes , author=. 2021 IEEE international conference on robotics and automation (ICRA) , pages=. 2021 , organization=
2021
-
[27]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Graspnet-1billion: A large-scale benchmark for general object grasping , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[28]
arXiv preprint arXiv:2507.13097 , year=
Graspgen: A diffusion-based framework for 6-dof grasping with on-generator training , author=. arXiv preprint arXiv:2507.13097 , year=
-
[29]
, author=
Finding locally optimal, collision-free trajectories with sequential convex optimization. , author=. Robotics: science and systems , volume=. 2013 , organization=
2013
-
[30]
arXiv preprint arXiv:2310.17274 , year=
curobo: Parallelized collision-free minimum-jerk robot motion generation , author=. arXiv preprint arXiv:2310.17274 , year=
-
[31]
2023 IEEE International conference on robotics and automation (ICRA) , pages=
Code as policies: Language model programs for embodied control , author=. 2023 IEEE International conference on robotics and automation (ICRA) , pages=. 2023 , organization=
2023
-
[32]
arXiv preprint arXiv:2307.05973 , year=
Voxposer: Composable 3d value maps for robotic manipulation with language models , author=. arXiv preprint arXiv:2307.05973 , year=
-
[33]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Genmanip: Llm-driven simulation for generalizable instruction-following manipulation , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.