REVIEW 4 major objections 3 minor 18 references
JENGA: Object selection and pose estimation for robotic grasping from a stack
T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A visual-inertial filter that keeps top-layer, unobstructed bricks reaches 0.79 mAP for single-brick grasping from a stack, versus 0.17 with no filtering
desk verdict A solid benchmark for stack grasping, but the ground truth labels partly mirror the method's own selection logic, so treat the headline numbers as self-consistent rather than physical. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the visibility ratio $r_i = |M^i_m| / |M^i_a|$, computed by rendering the full amodal silhouette of the estimated pose and dividing its pixel area by the visible segmentation mask, combined with a gravity-axis height rank obtained by projecting each object center into the global frame using the IMU acceleration reading. The two filters are composed as $F = F_v \circ F_h$, with visibility threshold $\epsilon_{vis}=0.80$ and top-$k$ height selection. On the benchmark side, a KD-tree and axis-aligned bounding-box IoU check labels a brick graspable when at least four of six principal directions are free (or three with no upper neighbor plus one free lateral and one free front-back direction), and evaluation uses Hungarian matching and average precision over ADD-S and MSSD threshold sets. These two quantities carry the argument.
What would settle it
Recompute the pipeline's mAP against graspability labels produced by a physical grasp simulator for the two-prong gripper, checking collisions, friction, and post-grasp stability instead of free-direction space; if mAP falls close to the vision-only or unfiltered values, the reported gain came from matching the labeler's own criterion rather than from true grasp readiness.
Extended reading notes
Core claim
The central claim is that object selection for stack grasping should be treated as a filtering problem on raw six-degree-of-freedom pose hypotheses rather than as a ranking by detector confidence. The paper's specific proposal is the composed filter $F = F_v \circ F_h$: first rank all detected bricks by their height along the IMU-measured gravity axis and keep the top $k$, then discard any brick whose visibility ratio $r_i = |M^i_m| / |M^i_a|$ falls below the threshold $\epsilon_{vis}=0.80$. On the synthetic benchmark with $k=1$, this yields 0.79 mAP over ADD-S and MSSD thresholds, versus 0.17 for all unfiltered poses, 0.60 for detector-confidence selection, 0.61 for inertial-only filtering, and 0.73 for vision-only filtering. The paper also claims that the problem remains genuinely hard: selecting two or three bricks from one image lowers mAP to 0.70 and 0.65, and a completely error-free solution is not achieved.
Load-bearing premise
The benchmark defines 'graspable' by a geometric rule, free space around a brick in at least four directions, and the method selects exactly the highly visible, top-layer bricks that this rule favors, so the evaluation may measure how well the filter matches its own definition rather than how often a real gripper succeeds.
Editorial extensions
If this is right
- A robot can select a single graspable brick from a stack with an RGB camera, an IMU, a detector, a pose estimator, and the composed filter, and the construction-site deployment is the paper's evidence for this.
- The filter is generic over the detector and pose estimator, so replacing either module with a more accurate one should improve the pipeline without changing the selection logic.
- The mAP drop from 0.79 for top $k=1$ to 0.70 for $k=2$ and 0.65 for $k=3$ implies the method is best suited to single-object picking, and multi-object picking from stacks remains open.
- Visibility filtering alone accounts for most of the gain (0.73 mAP) while inertial filtering alone gives 0.61, but the two are complementary and together add roughly six points of mAP.
Reading between the lines
- An independent grasp-planner label set, using collision meshes, gripper geometry, and friction, would test whether the free-direction rule is a good proxy for graspability; until then, the mAP numbers should be read partly as self-consistency between the selector and the labeler.
- The visibility-ratio idea transfers to any object category with amodal predictions, and a network that regresses $r_i$ directly from RGB could remove the need to render amodal masks at inference time.
- Because the IMU contributes only the gravity direction for height ranking, an image-only vertical-direction estimate from vanishing points or floor segmentation could make the method work without an IMU, at some cost to accuracy.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formally introduces the task of selecting an object to grasp from a structured stack and estimating its 6DoF pose. The proposed pipeline runs YOLOv8 detection, ZebraPose 6DoF estimation, a vision-based filter using the modal/amodal mask visibility ratio of Eq. (3) with threshold epsilon_vis=0.80, and an IMU-based gravity-axis height filter that keeps the top-k candidates. The authors contribute a BlenderProc synthetic brick-stack dataset with graspability labels generated by a free-neighbor-direction rule (Algorithm 1), an evaluation protocol that averages precision over ADD-S and MSSD pose thresholds, and ablations showing mAP 0.79 for top-k=1 versus 0.73 for vision-only and 0.61 for inertial-only filtering. A qualitative real-robot deployment on a brick pallet is shown in the supplementary video.
Significance. If the claims hold, the task definition and the dataset fill a genuinely under-explored gap in robotic manipulation of structured stacks, and the idea of combining a visibility mask with IMU-based height ranking is simple and practical. The evaluation metric that couples object selection with pose accuracy is a useful contribution to benchmarking this class of problems. However, the main quantitative claim currently rests on a synthetic benchmark whose ground-truth graspability labels are generated by a hand-coded geometric rule that closely resembles the method's own filters, and the real-world deployment is only shown qualitatively. These issues, together with the absence of error bars and sensitivity analysis, mean that the central claim is plausible but not yet convincingly established.
major comments (4)
- [V.B / Algorithm 1 vs. IV.D-IV.F] The ground-truth graspability label is generated by the free-neighbor rule in Algorithm 1, which labels a brick graspable if it has at least four missing principal directions (or, per the text, three with no upper neighbor and lateral clearance). This is essentially the same top-layer, unobstructed-object notion that the proposed visibility threshold (Eq. 3) and IMU height top-k filter (Section IV.E) implement. The mAP values in Tables I and II therefore largely measure how well the filter reproduces the labeling rule, not how often a grasp would physically succeed. To support the central claim, the authors should add an independent validation, for example real-robot grasp success rates on stacks, or relabel the benchmark with a physically motivated criterion such as collision-free force-closure grasps and show that the method ranking is unchanged.
- [Algorithm 1, lines 15-18] As printed, Algorithm 1 is not a faithful implementation of Section V.B. Line 15 increments m when every queried neighbor has positive IoU with the shifted bounding box, which counts a direction as blocked, whereas the text says a direction is missing precisely when no neighbor overlaps. The pseudocode also omits the three-missing-directions case with the additional structural constraints described in Section V.B. If this is a typesetting error, the benchmark code must be released and the labels regenerated; otherwise the ground-truth labels may be inverted.
- [IV.E, Eq. (7)] Equation (7), o_i,z_g = alpha_s o_s, is dimensionally inconsistent because alpha_s is an acceleration vector and o_s is a position vector in the sensor frame. The height projection should be the scalar product of the object position with the normalized gravity direction, for example (alpha_s / ||alpha_s||)^T o_s. The missing normalization and missing dot-product notation need to be corrected, and the implementation of the height ranking should be clarified, since this equation is load-bearing for the IMU-based filter.
- [Tables I and II / VI.C-VI.D] The quantitative evaluation reports single scalar mAP values without error bars, confidence intervals, or multiple seeds, despite the use of learned components and a 1K-image test split. The differences that motivate the combined filter are modest (mAP 0.79 vs. 0.73 vs. 0.61 in Table II), and without variance estimates it is unclear whether the combined method is significantly better than vision-only filtering. In addition, epsilon_vis=0.80 is fixed without a sensitivity analysis; a sweep over epsilon_vis and over top-k is needed to show the result does not depend critically on the chosen threshold.
minor comments (3)
- [VI.E] The deployment section refers only to a supplementary video. Please provide quantitative real-world measurements, such as the number of pick attempts and successes or pose errors against manual annotation, so that the real-world claim can be assessed.
- [V.A / VI.B] The dataset is described as being made available upon acceptance, but no URL or benchmark download mechanism is given. Please include a concrete release mechanism so that the benchmark is actually usable by other researchers.
- [IV.F] The composition notation F = F_v composed with F_h is used, but the accompanying sentence says the inertial filter is applied first and the vision filter second. Please state the order explicitly in both the notation and the text so there is no ambiguity about which filter is applied first.
Circularity Check
No significant circularity: the filtering pipeline is computed from RGB and IMU observations, and the benchmark labels are an external geometric rule that the filter does not use or fit.
full rationale
The claimed derivation chain is: raw pose hypotheses Π from YOLOv8+ZebraPose are passed through the visibility filter F_v (Eq. 4, threshold ε_vis=0.80) and the IMU height filter F_h (Eqs. 7-8), yielding G=F(Π,I,M,α_s). The evaluation then compares G to ground-truth graspability labels produced by Algorithm 1, which counts missing neighbors in six principal directions. Nothing in Eqs. 3-9 reads, uses, or is fitted to the Algorithm 1 labels: ε_vis is a stated fixed design choice, and top-k is an evaluation protocol, not a learned parameter. The two do share a common intuition (graspable objects are unobstructed and near the top of the stack), so the benchmark's external validity for physical grasping is limited and would benefit from real-robot success statistics; however, that overlap is a validity caveat, not a reduction of the prediction to the inputs. The filter could and does fail to recover the geometric rule (mAP 0.79 at top-k=1 is not 1.0, and baselines score much lower), which shows the evaluation is informative. The only self-citations ([1], [6]) are background references used to motivate the pose-estimation component; they are not load-bearing and no uniqueness theorem or ansatz is imported from the authors' prior work. The derivation is therefore self-contained with no circular step.
Assumptions & free parameters
free parameters (3)
- epsilon_vis =
0.80
- graspability rule thresholds =
missing >= 4 of 6 directions, or 3 with constraints
- top-k selection count =
k = 1, 2, 3
assumptions (5)
- domain assumption ZebraPose provides sufficiently accurate 6DoF pose hypotheses for the filtering to be meaningful.
- domain assumption The camera-IMU extrinsics T_sc are known and fixed.
- domain assumption Visibility ratio is a valid proxy for graspability of a brick.
- ad hoc to paper The synthetic BlenderProc dataset is representative of real construction brick stacks.
- ad hoc to paper Graspability can be defined by free-neighbor directions using axis-aligned bounding boxes.
Cite this review
Pith. "Pith review of JENGA: Object selection and pose estimation for robotic grasping from a stack." pith.science (2026). https://pith.science/paper/M2HWOXOB
@misc{pith2026250613425,
author = {Pith},
title = {Pith review of: JENGA: Object selection and pose estimation for robotic grasping from a stack},
year = {2026},
howpublished = {\url{https://pith.science/paper/M2HWOXOB}},
note = {Machine review of arXiv:2506.13425}
}
read the original abstract
Vision-based robotic object grasping is typically investigated in the context of isolated objects or unstructured object sets in bin picking scenarios. However, there are several settings, such as construction or warehouse automation, where a robot needs to interact with a structured object formation such as a stack. In this context, we define the problem of selecting suitable objects for grasping along with estimating an accurate 6DoF pose of these objects. To address this problem, we propose a camera-IMU based approach that prioritizes unobstructed objects on the higher layers of stacks and introduce a dataset for benchmarking and evaluation, along with a suitable evaluation metric that combines object selection with pose accuracy. Experimental results show that although our method can perform quite well, this is a challenging problem if a completely error-free solution is needed. Finally, we show results from the deployment of our method for a brick-picking application in a construction scenario.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Y . Lin, Y . Su, P. Nathan, S. Inuganti, Y . Di, M. Sundermeyer, F. Man- hardt, D. Stricker, J. Rambach, and Y . Zhang, “Hipose: Hierarchical binary surface encoding and correspondence pruning for rgb-d 6dof object pose estimation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 10 148–10 158
work page 2024
-
[2]
Bin picking approaches based on deep learning techniques: A state-of-the- art survey,
A. Cordeiro, L. F. Rocha, C. Costa, P. Costa, and M. F. Silva, “Bin picking approaches based on deep learning techniques: A state-of-the- art survey,” in2022 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC). IEEE, 2022, pp. 110– 117
work page 2022
-
[3]
In- troducing mvtec itodd-a dataset for 3d object recognition in industry,
B. Drost, M. Ulrich, P. Bergmann, P. Hartinger, and C. Steger, “In- troducing mvtec itodd-a dataset for 3d object recognition in industry,” inProceedings of the IEEE International Conference on Computer Vision Workshops, 2017, pp. 2200–2208
work page 2017
-
[4]
A method for registration of 3-d shapes,
P. Besl and N. D. McKay, “A method for registration of 3-d shapes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239–256, 1992
1992
-
[5]
M. Rad and V . Lepetit, “Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth,” inProceedings of the IEEE International Con- ference on Computer Vision (ICCV), 2017, pp. 3828–3836
work page 2017
-
[6]
Synpo-net—accurate and fast cnn-based 6dof object pose estimation using synthetic train- ing,
Y . Su, J. Rambach, A. Pagani, and D. Stricker, “Synpo-net—accurate and fast cnn-based 6dof object pose estimation using synthetic train- ing,”Sensors, vol. 21, no. 1, p. 300, 2021
work page 2021
-
[7]
Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,
Y . Xiang, T. Schmidt, V . Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” inRobotics: Science and Systems, 2018
work page 2018
-
[8]
Zebrapose: Coarse to fine surface encoding for 6dof object pose estimation,
Y . Su, M. Saleh, T. Fetzer, J. Rambach, N. Navab, B. Busam, D. Stricker, and F. Tombari, “Zebrapose: Coarse to fine surface encoding for 6dof object pose estimation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 6738–6748
work page 2022
Show all 18 references
-
[9]
Bop challenge 2023 on detection segmentation and pose estimation of seen and unseen rigid objects,
T. Hodan, M. Sundermeyer, Y . Labbe, V . N. Nguyen, G. Wang, E. Brachmann, B. Drost, V . Lepetit, C. Rother, and J. Matas, “Bop challenge 2023 on detection segmentation and pose estimation of seen and unseen rigid objects,” inProceedings of the IEEE/CVF Conference on Computer ...
2023
-
[10]
Data-driven grasp synthesis—a survey,
J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-driven grasp synthesis—a survey,”IEEE Transactions on Robotics, vol. 30, no. 2, pp. 289–309, 2014
2014
-
[11]
An overview of 3d object grasp synthesis algorithms,
A. Sahbani, S. El-Khoury, and P. Bidaud, “An overview of 3d object grasp synthesis algorithms,”Robotics and Autonomous Systems, vol. 60, no. 3, pp. 326–336, 2012, autonomous Grasping
2012
-
[12]
Robotic grasping from classical to modern: A survey,
H. Zhang, J. Tang, S. Sun, and X. Lan, “Robotic grasping from classical to modern: A survey,” 2022. [Online]. Available: https://arxiv.org/abs/2202.03631
2022 arXiv
-
[13]
A survey on learning-based robotic grasping,
K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber, “A survey on learning-based robotic grasping,”Current Robotics Reports, vol. 1, no. 4, pp. 239–249, 2020
2020
-
[14]
Ultralytics yolov8,
G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics yolov8,” 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
2023
-
[15]
Blenderproc2: A procedural pipeline for photorealistic rendering,
M. Denninger, D. Winkelbauer, M. Sundermeyer, W. Boerdijk, M. Knauer, K. H. Strobl, M. Humt, and R. Triebel, “Blenderproc2: A procedural pipeline for photorealistic rendering,”Journal of Open Source Software, vol. 8, no. 82, p. 4901, 2023
2023
-
[16]
Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,
S. Hinterstoisser, V . Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,” in11th Asian Conference on Computer Vision (ACCV). Springer, 2013, pp. 548– 562
2013
-
[17]
The hungarian method for the assignment problem,
H. W. Kuhn, “The hungarian method for the assignment problem,” Naval research logistics quarterly, vol. 2, no. 1-2, pp. 83–97, 1955
1955
-
[18]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in13th European conference on Computer Vision (ECCV). Springer, 2014, pp. 740–755
2014
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.