REVIEW 4 major objections 6 minor 1 cited by
RoboCup@Home 2024 OPL Winner NimbRo: Anthropomorphic Service Robots using Foundation Models for Perception and Planning
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper reports that a service robot combining open-vocabulary object segmentation with LLM-based task planning won the RoboCup@Home 2024 Open Platform League, showing that foundation models can replace much task-specific supervision in…
desk verdict Real competition win, externally scored; but the open-vocabulary generalization claim is not isolated from on-site tuning — a solid systems paper that needs clearer separation of verified outcome from broader claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing pairing is text-promptable grounding plus promptable segmentation: mmGrounding-DINO, an open-vocabulary detector that finds objects from text descriptions, outputs bounding boxes, and NanoSAM, a lightweight promptable segmenter, converts those boxes into instance masks. Masks are projected into depth to build a partial point cloud, optionally completed by registering a 3D model, and approximated by an oriented bounding box. Grasp poses are sampled on a quadrant sphere facing the object, filtered for collisions using a KD-tree over composite RGB-D and LiDAR point clouds, and ranked by a heuristic that favors precomputed collision-free pre-grasp poses with clearance from obstacles and the workspace boundary. Task planning is carried by an LLM that calls robot capability functions with textual feedback, allowing about 3 to 15 function calls per typical command.
What would settle it
Run the same robot in a fresh apartment using only generic object names with no on-site dataset and no prompt tuning; if it cannot segment and grasp a held-out set of household objects such as a mug, a sponge, and a pear with comparable success, the paper's claim that open-vocabulary approaches overcame labeling overhead is not supported.
Extended reading notes
Core claim
The authors report that open-vocabulary object segmentation is practically usable in a competitive household-robot setting: a text-promptable grounding detector (mmGrounding-DINO) produced bounding boxes for objects named in natural language, and a promptable segmenter (NanoSAM) refined those boxes into instance masks that fed the grasping pipeline. The same perception channel provided semantic scene information in the final demonstration, where the robot scanned an apartment, identified present objects, and later poured an egg into a pan. An LLM (GPT-4o) executed natural-language commands by calling a library of robot capability functions, advancing a task state machine until the command was accomplished or rejecting commands outside the robot's abilities. The authors state that this approach, together with a supervised detector for known object classes, won the Open Platform League with a total score of 8,852 points.
Load-bearing premise
The open-vocabulary generalization claim rests on the assumption that the manually designed text prompts and the on-site dataset used for tuning were not the decisive factor, since Section 3.3 says prompts were designed using locally captured data and could be changed on the fly, and additional data was collected for objects that performed poorly.
Editorial extensions
If this is right
- Household robots could be deployed to new environments without collecting and labeling a task-specific object dataset for every home, since text descriptions can stand in for training examples.
- LLM function calling with textual feedback can handle varied natural-language commands, including rejecting requests the robot cannot or should not perform, which matters for non-expert users.
- Combining closed-set detectors for known categories with open-vocabulary models for unknown items (shoes, socks, drinks, garbage) improves both precision and recall in monitoring tasks like Stickler for the Rules.
- The same perception pipeline carried over to the final demonstration, scanning a kitchen, supporting user-input-based planning, and executing a complex manipulation such as pouring an egg into a pan, suggesting reuse beyond predefined task stages.
Reading between the lines
- A natural extension the paper does not itself claim is that this architecture transfers to other domestic chores, such as tidying arbitrary objects or fetching items described by appearance, because the perception and planning layers are not tied to the competition's object classes.
- The on-site prompt and dataset tuning described in Section 3.3 suggests a testable boundary: if only generic prompts and no locally captured data are allowed, performance on unseen objects may drop, which would show that some of the claimed generalization is task-specific adaptation.
- The GPSR command about the smallest object on a counter hints that the same LLM-plus-vision loop could answer property questions (color, material, size) without a bespoke attribute classifier, by grounding attributes through text prompts and visual segments.
- A direct end-to-end measurement with prompts frozen at the start of a competition, rather than adjustable on the fly, would separate the contribution of the open-vocabulary models from the contribution of human prompt engineering.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports the approaches, hardware, and results of team NimbRo@Home at the RoboCup@Home 2024 Open Platform League, where the team won first place. The main technical emphasis is on open-vocabulary object segmentation and grasping, and on the use of large language models (LLMs) for natural language understanding and task planning. The paper describes the perception, planning, grasping, speech, and navigation modules, reports per-task competition outcomes and scores for the three stages and the final demonstration, and concludes that open-vocabulary segmentation proved valuable and that robustness and generalization were key to the win.
Significance. If the central claim is accepted, the paper demonstrates a practical integration of foundation models (open-vocabulary segmentation and LLM planning) into a deployed service robot that won a major competition. This would be a notable existence proof that promptable foundation models can reduce task-specific supervision in domestic robotics. The competition outcome itself is externally documented and the paper provides a useful system-level integration description, including a video reference and a comparison of final scores with other teams. However, the strength of the scientific contribution is limited by the absence of per-component quantitative evaluation and by the acknowledged on-site tuning of prompts and datasets, which weakens the generalization and robustness claims as stated.
major comments (4)
- [Section 3.3] The claim in the abstract that open-vocabulary segmentation and grasping of 'non-labeled objects' was demonstrated with 'robustness and generalization capabilities' is substantially weakened by the description of on-site tuning. The text states that prompts 'were manually designed using our locally captured dataset for evaluation and could be changed on the fly during the task' and that 'over the setup days, more data was collected for objects which were not performing well.' This means the open-vocabulary pipeline was adapted to the evaluation arena and its specific objects before and during the tests. The paper should either report which prompts were fixed a priori and which were changed, provide a held-out evaluation on objects not seen during the tuning days, or explicitly temper the generalization claim in the abstract and conclusion.
- [Section 4] The quantitative evidence consists solely of aggregate competition scores and single-run task anecdotes. No per-component metrics are reported, such as segmentation intersection-over-union, grasp success rates, number of grasp reattempts, LLM planning success rates, or a comparison between the open-vocabulary and closed-vocabulary pipelines on the same tasks. Consequently, the paper does not support the attribution of the competition win to the open-vocabulary and LLM components rather than to other subsystems (e.g., SLAM, person tracking, touchscreen fallback, closed-vocabulary detectors, or the overall system engineering). A table reporting per-task component success/failure counts and, where available, the scores from both arena runs would make the contribution of each component assessable.
- [Section 4.3] The final demonstration (egg pouring) is a single anecdote used to support the headline claim that open-vocabulary approaches can grasp non-labeled objects and execute complex tasks. Single demonstrations, without repeated trials, failure counts, or any quantitative measure of perception or manipulation success, do not support statements of robustness or generalization. The paper should report how many objects were scanned, how many grasps were attempted and succeeded, and how the open-vocabulary perception output was consumed by the task planner, or it should restrict the claim to a feasibility demonstration.
- [Section 5] The Lessons Learned section states that 'Open-vocabulary instance segmentation proved valuable in this competition,' but open-vocabulary models were used in only two stage tasks (Serving Breakfast and Clean the Table) and in combination with closed-vocabulary models in other tasks (e.g., Stickler for the Rules). Without a direct comparison or per-task attribution of the open-vocabulary contribution, this conclusion is not supported by the reported data. Please either provide such evidence or qualify the lesson to reflect the actual scope of open-vocabulary usage.
minor comments (6)
- [Section 2] The degree symbol in '180◦ FOV' should be typeset as '180°'; check for similar typographical issues throughout the paper.
- [Section 3.3] The phrase 'MeanIntersectionoverUnion' should be expanded with proper spacing as 'Mean Intersection over Union' for readability.
- [Section 3.6] The abbreviation 'SOTA' should be expanded on first use (e.g., 'state-of-the-art models') to make the text self-contained.
- [Section 4] The paper notes that 'the tests were executed twice in different arenas' but reports only a single aggregate score per stage. Reporting the per-run scores would give readers a better sense of variability.
- [Figure 4] The object perception pipeline diagram is visually dense; consider enlarging the figure or separating it into two panels for legibility.
- [References] Several reference entries for online resources (e.g., JACK Audio, Coqui TTS, Faster Whisper) lack access dates; add consistent access-date information where applicable.
Circularity Check
No significant circularity: the competition outcome is externally judged, and the paper's claims do not reduce to fitted parameters or self-citation chains.
full rationale
The paper's load-bearing result, winning the RoboCup@Home 2024 Open Platform League, is an externally evaluated competition outcome with independent judges and scores, not a quantity derived from the paper's own models or equations. The technical sections describe standard independently sourced components (SLAM Toolbox, YOLO, MaskDINO, mmGrounding-DINO, GPT-4o, cuRobo, etc.) and report actual task performance. Self-citations to prior NimbRo work are contextual and do not supply the evidence for this year's win. The closest issue is Section 3.3, where open-vocabulary prompts were manually designed using locally captured data and could be changed on the fly, and additional data was collected for poorly performing objects. That is a legitimate generalization confound for the 'open-vocabulary' claim, but it is not circularity: the paper does not define its prediction in terms of those inputs, nor does it fit a parameter and then rename the fit as a prediction. The paper's central outcome is measured against an external benchmark, so no derivation step reduces to its own inputs.
Assumptions & free parameters
assumptions (4)
- domain assumption The RoboCup@Home 2024 OPL rulebook and the judges' scores are a valid external benchmark for comparing service robots.
- domain assumption Pretrained off-the-shelf models (mmGrounding-DINO, NanoSAM, GPT-4o, Faster Whisper, cuRobo) behave on the robot as they do in their published descriptions.
- domain assumption Online access to the GPT-4o API is available and low-latency enough during task execution.
- ad hoc to paper Manually designed text prompts and on-site dataset curation are acceptable engineering inputs.
Cite this review
Pith. "Pith review of RoboCup@Home 2024 OPL Winner NimbRo: Anthropomorphic Service Robots using Foundation Models for Perception and Planning." pith.science (2026). https://pith.science/paper/LJYBL3SG
@misc{pith2026241214989,
author = {Pith},
title = {Pith review of: RoboCup@Home 2024 OPL Winner NimbRo: Anthropomorphic Service Robots using Foundation Models for Perception and Planning},
year = {2026},
howpublished = {\url{https://pith.science/paper/LJYBL3SG}},
note = {Machine review of arXiv:2412.14989}
}
read the original abstract
We present the approaches and contributions of the winning team NimbRo@Home at the RoboCup@Home 2024 competition in the Open Platform League held in Eindhoven, NL. Further, we describe our hardware setup and give an overview of the results for the task stages and the final demonstration. For this year's competition, we put a special emphasis on open-vocabulary object segmentation and grasping approaches that overcome the labeling overhead of supervised vision approaches, commonly used in RoboCup@Home. We successfully demonstrated that we can segment and grasp non-labeled objects by text descriptions. Further, we extensively employed LLMs for natural language understanding and task planning. Throughout the competition, our approaches showed robustness and generalization capabilities. A video of our performance can be found online.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Viser: Imperative, Web-based 3D Visualization in Python
The paper describes Viser, an open-source imperative, web-based 3D visualization library for Python with scene and GUI primitives.
Reference graph
Works this paper leans on
-
[1]
Journal of Field Robotics (JFR)36(1), 204–229 (2019)
Beul, M., Nieuwenhuisen, M., Quenzel, J., Rosu, R.A., et al.: Team NimbRo at MBZIRC 2017: Fast landing on a moving target and treasure hunting with a team of micro aerial vehicles. Journal of Field Robotics (JFR)36(1), 204–229 (2019)
work page 2019
-
[2]
Field Robotics2(1), 807–842 (2022)
Beul, M., Schwarz, M., Quenzel, J., Splietker, M., Bultmann, S., Schleich, D., Rochow, A., et al.: Target chase, wall building, and fire fighting: Autonomous UAVs of team NimbRo at MBZIRC 2020. Field Robotics2(1), 807–842 (2022)
work page 2022
-
[3]
In: IEEE- RAS 23rd International Conference on Humanoid Robots (Humanoids) (2024)
Bode, J., Pätzold, B., Memmesheimer, R., Behnke, S.: A comparison of prompt en- gineering techniques for task planning and execution in service robotics. In: IEEE- RAS 23rd International Conference on Humanoid Robots (Humanoids) (2024)
work page 2024
-
[4]
Bredin, H., Laurent, A.: End-to-end speaker segmentation for overlap-aware reseg- mentation. In: Interspeech Conference. pp. 3111–3115 (2021)
work page 2021
-
[5]
In: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG)
Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: VGGFace2: A dataset for recognising faces across pose and age. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG). pp. 67–74 (2018)
work page 2018
-
[6]
Davis, P., Letz, S.:https://jackaudio.org (2023)
work page 2023
-
[7]
Eren, G., Coqui TTS: Coqui TTS.https://github.com/coqui-ai/TTS (2021)
work page 2021
-
[8]
Hart, J., Moriarty, A., Pasternak, K., Kummert, J., Hawkin, A., Hassouna, V., Pena Narvaez, J.D., et al.: RoboCup@Home 2024: Rules and regulations (2024)
work page 2024
Show all 33 references
-
[9]
https://github.com/ ultralytics/ultralytics (2023)
Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLOv8. https://github.com/ ultralytics/ultralytics (2023)
2023
-
[10]
In: 38th International Conference on Ma- chine Learning (ICML)
Kim, J., Kong, J., Son, J.: Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In: 38th International Conference on Ma- chine Learning (ICML). pp. 5530–5540 (2021)
2021
-
[11]
In: IEEE/CVF International Conference on Computer Vision (ICCV)
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4015–4026 (2023)
2023
-
[12]
Field Robotics2(1), 55–74 (2022)
Lenz, C., Quenzel, J., Periyasamy, A.S., Razlaw, J., Rochow, A., Splietker, M., Schreiber,M.,Schwarz,M.,etal.:Autonomouswall-buildingandfirefighting:Team NimbRo’s UGV solution for MBZIRC 2020. Field Robotics2(1), 55–74 (2022)
2022
-
[13]
International Journal of Social Robotics (SORO) (2023) 12 R
Lenz, C., Schwarz, M., Rochow, A., Pätzold, B., et al.: NimbRo wins ANA Avatar XPRIZE immersive telepresence competition: Human-centric evaluation and lessons learned. International Journal of Social Robotics (SORO) (2023) 12 R. Memmesheimer et al
2023
-
[14]
In: IEEE/CVF Conf
Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L.M., et al.: Mask DINO: Towards a unified transformer-based framework for object detection and segmentation. In: IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) (2023)
2023
-
[15]
Journal of Open Source Software6(61), 2783 (2021)
Macenski, S., Jambrecic, I.: SLAM Toolbox: SLAM for the dynamic world. Journal of Open Source Software6(61), 2783 (2021)
2021
-
[16]
Science robotics7(66), eabm6074 (2022)
Macenski, S., Foote, T., Gerkey, B., et al.: Robot Operating System 2: Design, architecture, and uses in the wild. Science robotics7(66), eabm6074 (2022)
2022
-
[17]
RoboCup@Home Team Description Papers (2023)
Memmesheimer, R., Bode, J., Splietker, M., Bultmann, S., Imbusch, B.T., Behnke, S.: NimbRo@Home 2023 Open Platform League team description. RoboCup@Home Team Description Papers (2023)
2023
-
[18]
https://github.com/NVIDIA/ MAXINE-AFX-SDK (2023)
NVIDIA: NVIDIA MAXINE Audio Effects. https://github.com/NVIDIA/ MAXINE-AFX-SDK (2023)
2023
-
[19]
OpenAI: GPT-4 technical report (2023)
2023
-
[20]
Pages, J., Marchionni, L., Ferro, F.: TIAGo: The modular robot that adapts to different research needs. In: Int. WS on Robot Modularity, IROS. vol. 290 (2016)
2016
-
[21]
In: RoboCup 2023: Robot World Cup XXVI
Pavlichenko, D., Ficht, G., et al.: RoboCup 2023 Humanoid AdultSize winner Nim- bRo: NimbRoNet3 visual perception and responsive gait with waveform in-walk kicks. In: RoboCup 2023: Robot World Cup XXVI. Springer (2024)
2024
-
[22]
In: International Conference on Machine Learning (ICML)
Radford,A.,Kim,J.W.,Xu,T.,Brockman,G.,McLeavey,C.,Sutskever,I.:Robust speech recognition via large-scale weak supervision. In: International Conference on Machine Learning (ICML). pp. 28492–28518. PMLR (2023)
2023
-
[23]
Frontiers Robotics AI3, 57 (2016)
Schwarz, M., Beul, M., Droeschel, D., Schüller, S., Periyasamy, A.S., Lenz, C., Schreiber,M.,etal.:Supervisedautonomyforexplorationandmobilemanipulation in rough terrain with a centaur-like robot. Frontiers Robotics AI3, 57 (2016)
2016
-
[24]
Journal of Field Robotics36(1), 170–182 (2019)
Schwarz, M., Droeschel, D., et al.: Team NimbRo at MBZIRC 2017: Autonomous valve stem turning using a wrench. Journal of Field Robotics36(1), 170–182 (2019)
2019
-
[25]
Schwarz, M., Lenz, C., García, G.M., Koo, S., Periyasamy, A.S., Schreiber, M., Behnke, S.: Fast object learning and dual-arm coordination for cluttered stowing, picking, and packing. In: Int. Conf. on Robotics and Automation (ICRA) (2018)
2018
-
[26]
In: IEEE Int
Schwarz, M., Milan, A., Lenz, C., Munoz, A., Periyasamy, A.S., Schreiber, M., Schüller, S., Behnke, S.: NimbRo Picking: Versatile part handling for warehouse automation. In: IEEE Int. Conf. on Robotics and Automation (ICRA) (2017)
2017
-
[27]
Journal of Field Robotics (JFR)34(2), 400–425 (2017)
Schwarz, M., Rodehutskors, T., Droeschel, D., Beul, M., Schreiber, M., Araslanov, N., et al.: NimbRo Rescue: solving disaster-response tasks with the mobile manip- ulation robot Momaro. Journal of Field Robotics (JFR)34(2), 400–425 (2017)
2017
-
[28]
Serengil, S., Ozpinar, A.: A benchmark of facial recognition pipelines and co- usability performances of modules. J. Information Techn.17(2), 95–107 (2024)
2024
-
[29]
Frontiers Robotics AI3, 58 (2016)
Stückler, J., Schwarz, M., Behnke, S.: Mobile manipulation, tool use, and intuitive interaction for cognitive service robot Cosero. Frontiers Robotics AI3, 58 (2016)
2016
-
[30]
In: IEEE International Conference on Robotics and Automation (ICRA)
Sundaralingam, B., Hari, S.K.S., Fishman, A., Garrett, C., Van Wyk, K., et al.: cuRobo: Parallelized collision-free robot motion generation. In: IEEE International Conference on Robotics and Automation (ICRA). pp. 8112–8119 (2023)
2023
-
[31]
Systran: Faster Whisper.https://github.com/SYSTRAN/faster-whisper (2024)
2024
-
[32]
In: 20th International Conference on Information Fusion (FUSION)
Wojke, N., Memmesheimer, R., Paulus, D.: Joint operator detection and tracking for person following from mobile platforms. In: 20th International Conference on Information Fusion (FUSION). IEEE (2017)
2017
-
[33]
arXiv preprint arXiv:2401.02361 (2024)
Zhao, X., Chen, Y., Xu, S., Li, X., Wang, X., Li, Y., Huang, H.: An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361 (2024)
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.