REVIEW 4 major objections 5 minor 46 references
From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read A robot built from commodity hardware can autonomously stock supermarket shelves with 98% object-level reliability, though it remains ten times slower than human workers.
desk verdict A genuinely useful, honest systems paper on retail restocking; the 98.3% headline is plausible but not yet verifiable because the paper never pins down what counts as a success. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the 'last-meter' navigation stack: a 14-state Kalman filter fusing ArUco marker poses with a constant-velocity model feeds a two-step Model Predictive Controller that drives the robot base toward the shelf while steering the pan-tilt head to keep the marker in view. This solves the problem that the base lidar cannot see objects closer than 20 cm. Around that core, a Behavior Tree orchestrates docking, navigation, picking, and planogram-aware placement, while perception uses SAM2 segmentation, fine-tuned YOLO per-SKU classifiers, and an optional GPT-4o fallback for products without a trained model.
What would settle it
Run the same pipeline in a store aisle where items are spaced ~3 mm apart and include a wider range of package geometries; if object-level success drops materially below 98% for a comparable number of events, the central claim fails. A cheaper, paper-consistent check: count failures among products without a dedicated YOLO model—the paper itself predicts this is where all failures occur.
Extended reading notes
Core claim
The central claim is that a modular pipeline—behavior-tree task planning, ArUco-marker localization fused through a Kalman filter, a two-step model-predictive controller for base and head camera, and a perception stack pairing fine-tuned YOLO classifiers with SAM2 segmentation and tracking—can carry a Stretch 3 mobile manipulator through the full pick-navigate-place cycle in a supermarket-like environment with 98.3% object-level success over 724 events and a mean of 68.2 s per item. The paper further argues that most remaining failures come from products without a dedicated YOLO model, that task-level success reaches 87%, and that a joint cost-time index places autonomous stocking between te
Load-bearing premise
The 98% success rate depends on simplified products (cans and small boxes) and simplified packing (3 cm spacing instead of the ~3 mm common in stores); if those constraints are removed, the figure is not guaranteed by the paper's data.
Editorial extensions
If this is right
- If the 98.3% reliability transfers beyond the mock store, overnight restocking of canned and boxed goods is economically plausible: the robot's cost-time index is about a third of human performance but an order of magnitude better than teleoperation.
- The failure concentration among products lacking a dedicated YOLO model implies that expanding the per-SKU model library is the most direct route to higher task-level success.
- The 78%/22% hardware/software decomposition suggests that gripper and depth-sensing upgrades will shrink the speed gap more than any algorithmic change.
- The open-sourced modules and simulator give other groups a concrete baseline to test new perception or control components against the same 68.2 s/item and 98.3% figures.
Reading between the lines
- The reported experiments use 3 cm spacing between placed items, whereas real store shelves commonly pack items at about 3 mm; until the same pipeline is tested at real packing density, 98% should be read as an upper bound.
- The GPT-4o fallback's low accuracy and 6.9 s inference time, versus 0.2 s for fine-tuned YOLO, suggests an inexpensive testable improvement: an online mechanism that flags repeatedly-misclassified SKUs for model training, rather than relying on a generic vision-language model.
- If two or more of these robots were coordinated overnight in a single store, the architecture's task queue could scale throughput almost linearly; the paper does not address multi-robot collision or task allocation, but nothing in the reported design prevents that experiment.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an integrated autonomous supermarket stocking system built on the Hello Robot Stretch 3 mobile manipulator. The architecture combines ROS2, Behavior Trees, Nav2 with a spatio-temporal voxel layer, ArUco-based localization with a Kalman filter, a two-step MPC for base/head control, and a perception pipeline using YOLO and SAM2. The authors report laboratory experiments in a mock supermarket: 70 stocking tasks totaling 724 individual stocking operations, an object-level success rate exceeding 98% (and 98.3% in the discussion), a task-level success rate above 87%, and comparisons against teleoperation and human workers, including a joint time-cost performance index and a hardware/software gap decomposition. Several components are released as open-source ROS2 packages.
Significance. If the reported success rates are reproducible and well-defined, the paper would provide a useful baseline for an affordable, integrated retail stocking platform and a quantitative comparison against humans and teleoperation. The open-source release of the docking, navigation, and simulator packages is a concrete strength, as is the focus on a commercially available robot rather than custom high-end hardware. The paper is primarily a systems integration contribution, and the experimental evaluation is central to its claims.
major comments (4)
- [Sec. V.C–V.D] The central claim '98.3% object-level success, including pick and place operations and shelf navigation' is not verifiable as stated. No success rubric is defined: what counts as a successful stocking operation at the object level (placement tolerance, orientation, contact, gripper release, planogram compliance)? No raw success/failure counts by stage are given, and no confidence intervals are reported. The text says 8 of 9 unsuccessful tasks were partial failures due to 'minor placement inaccuracies' but does not say whether those inaccuracies are scored as object-level successes or failures. One task completely failed due to total loss of the ArUco marker; the number of objects in that task and whether they are in the 724 denominator are not stated. The phrase 'including pick and place operations and shelf navigation' is also unsupported: the experimental section reports individual sto
- [Sec. V.B] The reported 98% success rate is conditioned on a narrow product set and an artificial spacing requirement: 'Due to gripper size ... we focused on cans and small boxes' and 'placement required a spacing of 3cm between items, exceeding the common practice of approximately 3mm.' The paper also states that 'all failures involved products lacking a dedicated YOLO model,' meaning failures are concentrated in the GPT-4o-based perception branch. The claim that the system demonstrates 'reliable performance' for supermarket stocking is therefore broader than the evidence supports. The claims should be restricted to the tested product/spacing regime, or additional experiments under tighter spacing and with a more diverse product set should be provided.
- [Sec. V.D] The hardware/software gap decomposition—'approximately 78% of the lost performance should be attributed to hardware constraints and 22% to software limitations'—is asserted without a method. No equation, data table, sensitivity analysis, or error propagation is provided to show how these percentages are derived from the timing data. This decomposition is presented as one of the paper's contributions ('a hardware/software gap decomposition outlining concrete paths toward commercially viable retail robots'). It needs a transparent calculation or should be removed.
- [Sec. V.C] The statistical evaluation is incomplete. The paper reports point estimates (98% object-level, 87% task-level) over 70 tasks and 724 operations but gives no per-task variation, no confidence intervals, and no error bars in Fig. 7. The comparisons with Spahn et al. [6] and Wu et al. [11] cite their success rates without adjusting for different protocols or task definitions. Please add confidence intervals for the proportions (e.g., Clopper-Pearson) and clearly state the number of objects per task and the definition of task success, so the headline numbers are interpretable and comparable.
minor comments (5)
- [Introduction] The text says 'Finally, Section IV presents extensive laboratory tests' but the experiments are presented in Section V. The section numbering is inconsistent.
- [Sec. V.D] The heading 'Performace' should be 'Performance'.
- [Algorithm 1] Typos in function names: 'Get_Bouning_Boxes' appears twice; should be 'Get_Bounding_Boxes'.
- [Eq. (13)] The performance index 'pi = 1000 / (Annual Cost · Time Per Item)' would benefit from explicit units and a definition of Annual Cost (USD/year) and Time Per Item (seconds). The text uses p_i,AR, p_i,H, p_i,T without defining the subscript convention.
- [Sec. III.A] Minor typo: 'unicyle' should be 'unicycle'.
Circularity Check
No significant circularity: the central results are empirical measurements, and the cited self-work is not load-bearing.
full rationale
The paper's headline quantitative claims—98.3% object-level success over 724 stocking operations and 68.2 s per item—are empirical measurements from the integrated system, not derived quantities in which an output is recycled from an input. The control and estimation components (KF, two-step MPC, visual servo gains) use standard textbook formulations (Secs. II-B, II-C, IV-C), with parameters tuned from logs or trial-and-error; the reported success rates are subsequent independent experiments, so no fitted parameter is renamed as a prediction. The only self-citations are [2], an introductory reference for multi-robot agriculture that is not load-bearing, and [35], cited for an ArUco field-of-view safety constraint in the head MPC (Sec. IV-C.2, near Eq. (12)). Even if [35] is by a co-author, it is used as a constraint reference, not as a uniqueness theorem or as the source of the success claim. The paper's stated limitations—3 cm spacing, canned/small-box focus, and failures concentrated on products without dedicated YOLO models—affect external validity and metric interpretability, but they do not make the experimental results circular. I find no step where a claimed result is equivalent by construction to an input or where a prediction is statistically forced by a fitted parameter.
Assumptions & free parameters
free parameters (5)
- Docking visual servoing gains K_x, K_y =
0.75, -5.0
- Manipulation velocity gains P_b, P_l, P_a =
0.0005, 0.003, 0.9
- KF noise covariances sigma_x, sigma_z =
0.5
- MPC weights Q, R, K and horizon N
- YOLO_grocery training hyperparameters =
200 epochs, lr 0.005 to 0.0001
assumptions (6)
- domain assumption The mock supermarket with pre-mapped layout, known cart position, and no human bystanders is representative of the intended overnight deployment scenario.
- domain assumption Products are limited to cans and small boxes that fit the 2.5 cm gripper and can be spaced at 3 cm (or about 5 mm for cylinders).
- domain assumption Known GTIN product dimensions and planogram locations are available and correct.
- domain assumption ArUco markers installed in shelf label rails have known world-frame poses and remain visible; if lost, task fails.
- standard math Standard linear Kalman filter and MPC formulations with Gaussian noise and constant-velocity models are appropriate for the pose-estimation and navigation problem.
- domain assumption Cost model assumptions: USD 25,000 robot, 5,000-hour lifespan, 8-hour shifts, 2 hours weekly maintenance, and literature-based human labor costs.
Cite this review
Pith. "Pith review of From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator." pith.science (2026). https://pith.science/paper/JYBFV6EQ
@misc{pith2026250911740,
author = {Pith},
title = {Pith review of: From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator},
year = {2026},
howpublished = {\url{https://pith.science/paper/JYBFV6EQ}},
note = {Machine review of arXiv:2509.11740}
}
read the original abstract
Autonomous stocking in retail environments, particularly supermarkets, presents challenges due to dynamic human interactions, constrained spaces, and diverse product geometries. This paper introduces an efficient modular robotic system for autonomous shelf stocking, integrating commercially available hardware with a scalable algorithmic architecture. A major contribution of this work is the system integration of off-the-shelf hardware and ROS2-based perception, planning, and control into a single deployable platform for retail environments. Our solution leverages Behavior Trees (BTs) for task planning, fine-tuned vision models for object detection, and a two-step Model Predictive Control (MPC) framework for precise shelf navigation using ArUco markers. Laboratory experiments replicating realistic supermarket conditions demonstrate reliable performance, achieving over 98% success in pick-and-place operations across a total of more than 700 stocking events. However, our comparative benchmarks indicate that the performance and cost-effectiveness of current autonomous systems remain inferior to that of human workers, which we use to highlight key improvement areas and quantify the progress still required before widespread commercial deployment can realistically be achieved.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[6]
Demonstrating adap- tive mobile manipulation in retail environments,
M. Spahn, C. Pezzato, C. Salmi, R. Dekker, C. Wang, C. Pek, J. Kober, J. Alonso-Mora, C. H. Corbato, and M. Wisse, “Demonstrating adap- tive mobile manipulation in retail environments,”Proceedings of the Robotics: Science and System XX, 2024
2024
-
[11]
In the wild ungraspable object picking with bimanual nonprehensile manipulation,
A. Wu and D. Kruse, “In the wild ungraspable object picking with bimanual nonprehensile manipulation,” 2024. [Online]. Available: https://arxiv.org/abs/2409.15465
arXiv 2024
-
[1]
Robotics in manufacturing: A review of advances in automation and workforce implications,
R. Adebayo, C. Nwankwo, I. Festus-Ikhuoria, and O. Olajiga, “Robotics in manufacturing: A review of advances in automation and workforce implications,”International Journal of Advanced Multidis- ciplinary Research and Studies, vol. 4, pp. 632–638, 03 2024
2024
-
[2]
Efficient coordination and synchronization of multi- robot systems under recurring linear temporal logic,
D. Peron, V . N. Fernandez-Ayala, E. E. Vlahakis, and D. V . Di- marogonas, “Efficient coordination and synchronization of multi- robot systems under recurring linear temporal logic,” in2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 10 194–10 200
2025
-
[3]
53-7065 stockers and order fillers,
U.S. Bureau of Labor Statistics, “53-7065 stockers and order fillers,” Mar. 2024, accessed: 2025-03-24. [Online]. Available: https://www.bls.gov/oes/2023/may/oes537065.htm
2024
-
[4]
Forecasting workforce for store attendants in a grocery retailer,
E. Z. Güney, “Forecasting workforce for store attendants in a grocery retailer,”KNOWLEDGE-International Journal, vol. 31, no. 1, pp. 293– 297, 2019
2019
-
[5]
Dhaliwal,The Rise of Automation and Robotics in Warehouse Management
A. Dhaliwal,The Rise of Automation and Robotics in Warehouse Management. CRC Press, 10 2020, pp. 63–72
2020
-
[7]
Shelf replenishment based on object arrangement detection and collapse prediction for bimanual manipulation,
T. Motoda, D. Petit, T. Nishi, K. Nagata, W. Wan, and K. Harada, “Shelf replenishment based on object arrangement detection and collapse prediction for bimanual manipulation,”Robotics, vol. 11, no. 5, 2022
2022
Show all 46 references
-
[8]
An investigation towards an efficient algorithm for the automation of stocking and disposal tasks in a convenience store using a robotic arm system,
F. Sasono and K. Wada, “An investigation towards an efficient algorithm for the automation of stocking and disposal tasks in a convenience store using a robotic arm system,”Journal of Robotics and Mechatronics, vol. 37, pp. 231–239, 02 2025
2025
-
[9]
Bossa nova robotics,
“Bossa nova robotics,” https://www.bossanova.com/bossa-nova
-
[10]
Practical insights on grasp strategies for mobile manipulation in the wild,
I. Huang, R. Cheng, S. Kim, D. Kruse, C. Matl, L. Kaul, J. Hancock, S. Harikumar, M. Tjersland, J. Borders, and D. Helmick, “Practical insights on grasp strategies for mobile manipulation in the wild,”
-
[12]
Can robots refill a supermarket shelf?: Motion planning and grasp control,
M. Costanzo, G. De Maria, G. Lettera, and C. Natale, “Can robots refill a supermarket shelf?: Motion planning and grasp control,”IEEE Robotics & Automation Magazine, vol. PP, pp. 2–14, 03 2021
2021
-
[13]
Towards autonomous retail stocking and picking: Methods enabling robust vacuum-based robotic manipulation in densely packed environments,
P. Kmecl, M. Munih, and J. Podobnik, “Towards autonomous retail stocking and picking: Methods enabling robust vacuum-based robotic manipulation in densely packed environments,”Sensors, vol. 24, no. 20, 2024
2024
-
[14]
Shelf management: A deep learning-based system for shelf visual monitoring,
R. Pietrini, M. Paolanti, A. Mancini, E. Frontoni, and P. Zingaretti, “Shelf management: A deep learning-based system for shelf visual monitoring,”Expert Systems with Applications, vol. 255, p. 124635,
-
[15]
An end-to-end computationally lightweight vision-based grasping system for grocery items,
T. Mansakul, G. Tang, P. Webb, J. Rice, D. Oakley, and J. Fowler, “An end-to-end computationally lightweight vision-based grasping system for grocery items,”Sensors, vol. 25, no. 17, 2025. [Online]. Available: https://www.mdpi.com/1424-8220/25/17/5309
2025
-
[16]
Hello robot stretch 3,
“Hello robot stretch 3,” https://hello-robot.com/stretch-3-product
-
[17]
You only look once: Unified, real-time object detection,
“You only look once: Unified, real-time object detection,” in2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788
2016
-
[18]
Sam 2: Segment anything in images and videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Dollár, and C. Feichtenhofer, “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408....
2024 arXiv
-
[19]
Automatic generation and detection of highly reliable fiducial markers under occlusion,
S. Garrido-Jurado, R. M. noz Salinas, F. Madrid-Cuevas, and M. Marín-Jiménez, “Automatic generation and detection of highly reliable fiducial markers under occlusion,”Pattern Recognition, vol. 47, no. 6, pp. 2280–2292, 2014. [Online]. Available: https: //www.sciencedirect.com/...
2014
-
[20]
A new approach to linear filtering and prediction problems,
R. E. Kalman, “A new approach to linear filtering and prediction problems,”Journal of Basic Engineering, vol. 82, no. 1, p. 35, 1960. [Online]. Available: http://dx.doi.org/10.1115/1.3662552
1960 doi
-
[21]
Rawlings, D
J. Rawlings, D. Mayne, and M. Diehl,Model Predictive Control: Theory, Computation, and Design. Nob Hill Publishing, 2017
2017
-
[22]
The marathon 2: A navigation system,
S. Macenski, F. Martin, R. White, and J. Ginés Clavero, “The marathon 2: A navigation system,” in2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020
2020
-
[23]
https://github.com/animumai/ros2_stretch_docking
-
[24]
https://github.com/animumai/stretch_nav2_offloaded
-
[25]
https://github.com/animumai/ros2_iot_bridge
-
[26]
https://github.com/animumai/ros2_simulator_stretch
-
[27]
Behavior trees in robotics and ai,
M. Colledanchise and P. Ögren, “Behavior trees in robotics and ai,” Jul
-
[28]
https://docs.ultralytics.com/models/yolo11
-
[29]
Ros2 - robot operating system 2,
“Ros2 - robot operating system 2,” https://www.ros.org/
-
[30]
Banks, E
A. Banks, E. Briggs, K. Borgendale, and R. Gupta,MQTT V ersion 5.0, OASIS Open Std., 03 2019, oASIS Standard. [Online]. Available: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html
2019
-
[31]
https://docs.nav2.org/tutorials/docs/navigation2_with_stvl.html
-
[32]
Aggressive driving with model predictive path integral control,
G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Aggressive driving with model predictive path integral control,” in 2016 IEEE International Conference on Robotics and Automation (ICRA), 2016, pp. 1433–1440
2016
-
[33]
https://github.com/SteveMacenski/slam_toolbox
-
[34]
Visual servo control. i. basic approaches,
F. Chaumette and S. Hutchinson, “Visual servo control. i. basic approaches,”Robotics & Automation Magazine, IEEE, vol. 13, pp. 82 – 90, 01 2007
2007
-
[35]
Robust visual servoing under human supervision for assembly tasks,
V . N. Fernandez-Ayala, J. Silva, M. Guo, and D. V . Dimarogonas, “Robust visual servoing under human supervision for assembly tasks,” 2025. [Online]. Available: https://arxiv.org/abs/2504.12506
2025 arXiv
-
[36]
Solving automatic check-out with fine-tuned yolo models,
R. Skinderowicz, “Solving automatic check-out with fine-tuned yolo models,”Procedia Computer Science, vol. 246, pp. 1649– 1658, 2024, 28th International Conference on Knowledge Based and Intelligent information and Engineering Systems (KES 2024). [Online]. Available: https://w...
2024
-
[37]
GS1 General Specifications,
GS1 AISBL, “GS1 General Specifications,” GS1 AISBL, Brussels, Belgium, Tech. Rep. Version 25.0, January 2025, the foundational GS1 standard defining identification keys (including GTIN/EAN), data attributes, application rules, and data carrier specifications for the global GS1...
2025
-
[38]
Gpt-4 technical report,
OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, and B. Zoph, “Gpt-4 technical report,” 03 2023
2023
-
[39]
Inverse kinematics on stretch,
H. Robot, “Inverse kinematics on stretch,” https://github.com/hello-r obot/stretch_tutorials/blob/master/stretch_body/jupyter/inverse_kine matics.ipynb, 03 2023
2023
-
[40]
Fronta au- tonomous stocking,
D. Peron, V . Nan Fernandez-Ayala, and L. Segelmark, “Fronta au- tonomous stocking,” Youtube. [Online]. Available: https://www.yout ube.com/watch?v=D6nUanEhqNc
-
[41]
Ros2 gazebo simulation
C. U. C. Robotics, “Ros2 gazebo simulation.” [Online]. Available: https://github.com/CardiffUniversityComputationalRobotics/stretch_r os2/tree/humble/stretch_gz_sim
-
[42]
The supermarket, by department,
Food Marketing Institute (FMI), “The supermarket, by department,” Web page, 2011, accessed: 2025-09-13. [Online]. Available: https: //www.fmi.org/docs/facts-figures/grocerydept.pdf
2011
-
[43]
Table 4. employer costs for employee compensation for private industry workers by occupational and industry group,
U. B. of Labor Statistics, “Table 4. employer costs for employee compensation for private industry workers by occupational and industry group,” 2024, accessed: 2025-04-07. [Online]. Available: https://www.bls.gov/news.release/ecec.t04.htm
2024
-
[2018]
Available: http://dx.doi.org/10.1201/9780429489105
[Online]. Available: http://dx.doi.org/10.1201/9780429489105
-
[2024]
Available: https://www.sciencedirect.com/science/arti cle/pii/S0957417424015021
[Online]. Available: https://www.sciencedirect.com/science/arti cle/pii/S0957417424015021
-
[2025]
Available: https://arxiv.org/abs/2504.12512
[Online]. Available: https://arxiv.org/abs/2504.12512
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.