{"id":"95e571b3-d910-4665-8c9b-1fc789709ba7","arxiv_id":"2504.12512","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a real grocery store, the SHOPPER robot achieved 61 percent success across 335 pick attempts and attributes all failures to eight root causes.","lead":"Toyota Research Institute's SHOPPER robot picked items in a real grocery store and succeeded 61 percent of the time. Its failure analysis identifies the main reasons picks fail, helping robotics teams decide where to focus.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I's root-cause distribution rests on unvalidated post-hoc manual labeling; without a protocol or per-pick evidence, the eight-cause taxonomy is the weakest link in the paper's central insight.","rationale":"I read the paper as a systems report whose central value is the honest, large-scale field failure analysis. The 61% over 335 distinct picks is a credible self-reported operational metric, and the paper is careful to define success as the target item being placed in the basket. The robot's decision to skip items that are not in sight means the rate is conditional on detection, but the strongest claim explicitly says 'attempted picks,' so this is not a misrepresentation. The reader's identified weakest assumption—unvalidated manual labeling of failure causes—is indeed the most load-bearing concern. The taxonomy is not a minor descriptive addition; it is the primary contribution and the basis for the paper's recommendations about state tracking, dual-arm coordination, and perception. The paper also provides no confidence intervals around either the overall success rate or the per-cause percentages, and the sample sizes behind several rows are small (e.g., R8 at 4.0% of 131 failures is about five picks). However, this is a completeness issue that data release would address, not a demonstrated flaw in the experiments. The proposed test—independent annotation of released per-pick evidence with a pre-registered protocol—directly settles whether the taxonomy is reproducible or subjective. If the test passes, the taxonomy is substantially strengthened; if it fails, the paper's main claim would need to be downgraded to a description of the authors' internal debugging categories rather than a validated set of eight root causes. I therefore see no reason to change the reader's conditional verdict, and I agree with the reader that data release and a labeling protocol are the appropriate conditions.","tokens_in":9864,"tokens_out":4200,"duration_ms":49254,"concrete_test":"Release the recorded per-pick sensor data (videos and, where available, point clouds) for all 131 failures along with a pre-registered labeling protocol that defines each root cause by observable criteria. Have at least two independent annotators, blind to the paper's reported distribution, label every failed pick. Compute inter-annotator agreement (e.g., Cohen's kappa) and compare per-category rates against Table I. If kappa is below 0.6, or if any category's rate shifts by more than 5 percentage points, the root-cause ranking and the associated insights are not supported by the evidence as presented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main scientific contribution is not the 61% point estimate but the claim that all pick failures reduce to eight fundamental root causes with the specific distribution shown in Table I (R1 36.0%, R2 15.2%, R3 14.4%, etc.). This distribution is the evidence behind the paper's practical insights and future-work priorities, yet it is produced by the authors' own post-hoc labeling of each failed pick. The paper reports no labeling protocol, no definition of when a failure is assigned to R1 versus R2 versus R7, no inter-annotator agreement, and no per-pick video or annotation artifact. The categories themselves are partly overlapping in causal terms: R3 (displaced item), R5 (neighboring interference), and R6 (fell en route) are all described in Section VI.B as consequences of the same underlying lack of real-time item state tracking, so a single observed failure could plausibly be coded into more than one bucket. Because the failure percentages drive the qualitative conclusions, unvalidated coding is a load-bearing threat to the central claim. The 61% success rate itself is a self-reported point estimate with no confidence interval, but the taxonomy is the more consequential uncertainty; the paper explicitly frames 'eight fundamental root causes' as a primary contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents SHOPPER, a wheeled dual-arm mobile manipulation platform with suction and gripper tools, and reports on its deployment in an unmodified grocery store. The authors describe a modular pipeline (perception, grasp-strategy classification, grasp pose planning, execution with admittance control) and a set of grasp strategies for shelves, produce piles, handled items, bags, and obstacles. The central empirical result is a field test with 335 distinct pick attempts and a 61% success rate (item placed in basket). The paper's main analytical contribution is a taxonomy of eight root causes (R1-R8) for all observed failures, with the distribution reported in Table I. The authors then draw practical conclusions about bottlenecks such as heavy/bulky items, open-loop execution, lack of real-time item state tracking, difficult extractions, and perception/kinematic noise.","tokens_in":10103,"tokens_out":5045,"duration_ms":54610,"significance":"If the failure taxonomy and success-rate claims are taken at face value, this is a useful systems-level data point for the mobile manipulation community: it quantifies performance in a genuinely unstructured environment, describes concrete engineering mitigations (dithering, admittance control with a non-diagonal stiffness term, tool-pose correction, ICP alignment), and identifies open problems. The paper's strengths include its full-stack online operation in an unmodified store, the relatively large number of pick attempts, the explicit disclosure of excluded glass items, and the public website with per-trial rankings. The main weakness is that the central analytical contribution, the R1-R8 distribution in Table I, rests on a post-hoc manual labeling process that is not described, validated, or made auditable; because the percentages drive the paper's qualitative insights, this is a load-bearing issue rather than a presentation concern.","major_comments":[{"comment":"The paper reports R1-R8 as \"eight fundamental root causes behind all pick failures\" and uses Table I to rank their frequencies, but no labeling protocol is provided. There are no explicit definitions of the decision boundary between categories (for example, when a failure is R3 versus R5 versus R6), no statement of whether labels were assigned prospectively or retrospectively by one or multiple coders, no per-pick evidence, and no inter-annotator agreement measure. Since the percentages (R1 at 36.0%, R2 at 15.2%, R3 at 14.4%) drive the conclusions in Section VI.B, please provide a coding protocol with mutually exclusive decision rules, raw counts per category and per tool, and, if possible, per-pick annotations or a reliability check. Without this, the distribution in Table I should be described as an exploratory classification rather than a validated measurement.","section":"Section VI.A / Table I"},{"comment":"Several of the eight categories are causally overlapping as discussed in the text. R3 (unexpectedly displaced target item), R5 (interference from neighboring item), and R6 (fell on the way to basket) are all attributed in Section VI.B to the same underlying deficiency, namely the lack of real-time item state tracking. Similarly, R4 (difficult extract) and R2 (edge cases in planning algorithms) can be hard to separate when shelf geometry restricts the admissible extraction trajectory. As defined, a single observed failure could plausibly be assigned to more than one bucket, so the reported percentages do not necessarily form a partition. Please either define mutually exclusive coding criteria that resolve these overlaps or reframe the table as symptom categories rather than fundamental root causes.","section":"Section VI.B / Table I"},{"comment":"The success-rate accounting is ambiguous with respect to recovery actions. Section V.B states that during the latest field test, 14 of 17 instances in which the suction grasp initially failed were successfully recovered through dithering. It is not specified whether these recovered attempts are counted as successes or as failures in the 61% value, nor whether the 17 instances are included in the 335 distinct pick attempts. Please define precisely what counts as a pick attempt, what counts as a success, and how transient failures that are later recovered are recorded in the denominator and in Table I.","section":"Section V.B / Section VI.A"},{"comment":"Table I presents percentages without raw counts or uncertainty. With 335 picks and a 39% failure rate, the rarest categories correspond to roughly five to ten observed failures (e.g., R8 at 4.0% overall, or R8 at 2.5% for suction), and small changes in coding can move these percentages substantially. Tool-specific comparisons are correspondingly fragile. Please report the raw number of failures per category and per tool, and provide at least the denominators or binomial confidence intervals so that the reader can assess the precision of the ranking.","section":"Section VI.A / Table I"}],"minor_comments":[{"comment":"The abstract says the goal is \"picking any item in a real grocery store,\" but Section IV.A discloses that 236 of 959 automatically detected items are excluded from the graspable set because they are made of glass. Please qualify the \"any item\" phrasing to match the actual item scope.","section":"Abstract / Section IV.A"},{"comment":"The non-diagonal admittance control term is described only verbally. An equation or a small schematic showing how a deviation along the extraction axis produces motion in the vertical axis would make the mechanism much easier for readers to reproduce.","section":"Section V.C"},{"comment":"The tool-specific columns do not all sum exactly to 100% (the gripper column sums to approximately 99.9%), presumably due to rounding. Please add a footnote stating that percentages are rounded and indicate the total number of failures per tool.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper is a field-systems report rather than a methods paper, and its main evidence is the 335-pick field test. I found no indication of fabrication; the count-based success rate is plausible. The central issue is that the failure taxonomy, which is the paper's main analytical contribution, is not currently auditable. The requested revisions (coding protocol, raw counts, uncertainty, and a clarified success accounting) are within the scope of the manuscript and should be feasible without new experiments. I would not recommend rejection, but the taxonomy must be strengthened before the paper can serve as a reliable reference for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the empirical failure analysis—335 picks in an unmodified grocery store with a breakdown into eight root causes. The grasp strategies themselves are mostly assembled from known pieces and the authors' prior RSS 2023 system, but the field data is new and genuinely useful. The stress-test note is on target: the failure taxonomy is built on the authors' post-hoc labeling of each failed pick, with no coding protocol, no inter-annotator agreement, and no per-pick evidence in the paper. That makes the specific percentages in Table I directional, not measured. The central success-rate claim (61%) is a simple count and I find it credible. The eight-cause list is plausible and matches my intuition about where real-world grasping breaks: heavy/bulky items, open-loop execution, extraction clearance, perception noise. The categories overlap in causal terms—R3, R5, and R6 all stem from lack of real-time item state tracking—so a single failure might plausibly be coded into more than one bucket. The paper would be stronger if it shipped the per-pick annotations, the labeling protocol, and ideally inter-annotator agreement numbers. Without those, the taxonomy is a useful framing but not an objective measurement. I do not see a load-bearing flaw in the system's engineering: the description of the grasp pipeline, the non-diagonal admittance coupling for extraction, and the dithering recovery data (14/17) are all concrete and credible. The paper is honest about excluding glass items for safety; that's a limitation to keep in mind, not a flaw. The citation pattern is fine—the authors cite their prior work for system details, and the empirical result is self-contained. This paper is for robotics practitioners working on mobile manipulation and grasping, especially those doing field deployments. It deserves a serious referee: the data is valuable, the insights are actionable, and the writing is clear. The referee should ask for the annotation protocol, per-pick evidence, and confidence intervals for the percentages. My recommendation is to engage with it—conditional acceptance is appropriate if the authors can show the taxonomy isn't just one team's subjective post-hoc coding.","headline":"The failure analysis is the real contribution, but the taxonomy's percentages rest on unvalidated post-hoc labeling, so treat them as directional, not measured.","tokens_in":10640,"tokens_out":3495,"would_cite":true,"duration_ms":34175,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that a fully autonomous mobile manipulator, SHOPPER, succeeded in 61 percent of 335 distinct grocery picks in an unmodified store, and attributes every failure to one of eight root causes.","keywords":["mobile manipulation","grasping","grocery store","field testing","failure analysis","suction grasp","admittance control","real-world deployment"],"falsifier":"Re-annotate the recorded failed picks with an independent panel using a written protocol and check whether the R1-R8 distribution is reproduced; if coders disagree or the top cause changes, the reported percentages are not stable ground truth.","tokens_in":9667,"feed_emoji":"🛒","tokens_out":10691,"duration_ms":106672,"temperature":0.7,"pith_summary":"This paper reports a large-scale field test of SHOPPER, a mobile manipulator that autonomously retrieves grocery items in an unmodified real store. The authors aim to establish where the bottlenecks actually are for in-the-wild grasping: after 335 distinct pick attempts and about ten hours of runtime, SHOPPER succeeded 61 percent of the time, and every failure was sorted into one of eight root causes. The largest class, at 36 percent, is items that are too heavy, bulky, or otherwise incompatible with the available tools; the paper argues that open-loop execution, missing real-time item state tracking, and immature bimanual coordination are the fundamental gaps. The value of the claim is practical: it turns vague talk about real-world grasping difficulty into a quantified failure distribution that can direct future research.","feed_headline":"61% success in 335 grocery picks; eight failure causes","feed_subtitle":"After 10 hours in a real grocery store, the robot's 335 picks show what still blocks reliable grasping.","key_machinery":"The argument is carried by the SHOPPER platform and its grasp-strategy suite operating as one integrated system, plus the eight-cause failure taxonomy that organizes the field-test data. The hardware pairs a two-finger gripper with a custom suction tool; the software pipeline detects and segments items, classifies them into strategy categories, plans grasps with either 2D antipodal hull sampling, 3D ellipsoidal shape-completed sampling, handle keypoint insertion, or side grasps for bags, and executes through compliant admittance control. Three mechanisms do the heaviest lifting: dithering, small compliant probing motions that recovered 14 of 17 initially failed suction grasps; a non-diagonal stiffness coupling in the admittance controller that deflects the tool upward when extraction is blocked by a shelf lip; and runtime tool-pose correction by point-cloud registration that brings commanded gripper poses to sub-centimeter accuracy. The taxonomy R1-R8 is what turns the raw pick attempts into a conclusion: each failed pick is assigned to one root cause, and the resulting distribution is the paper's evidence for where the remaining research effort should go.","core_discovery":"On its own terms, the paper's central discovery is a quantified failure taxonomy from a fully integrated robot operating in a real grocery store. SHOPPER attempted 335 distinct picks across three nights and successfully placed the target item in its basket in 61 percent of attempts. The authors attribute all observed failures to eight root causes: R1 heavy, bulky, or incompatible items (36.0 percent); R2 edge cases in motion and grasp planning (15.2 percent); R3 the target item being unexpectedly displaced during execution (14.4 percent); R4 difficult extraction from fixtures (8.8 percent); R5 interference from neighboring items (8.8 percent); R6 the item falling off the tool en route to the basket (7.2 percent); R7 perception errors such as missed collision voxels (5.6 percent); and R8 kinematic imprecision (4.0 percent). The tool-specific breakdown shows that suction failures are dominated by item incompatibility, while gripper failures are dominated by unexpected item displacement. The paper's interpretive claim is that most of these failures trace to open-loop execution without real-time item state tracking, and that the capabilities most needed for general-purpose mobile manipulation are reliable object state tracking and dual-arm coordination.","pith_inferences":["Extending the paper's data, the non-planning failure classes look addressable by hardware and control changes; if they are additive, eliminating the largest tool-relevant classes could plausibly lift success from 61 percent to the mid-80s on this item set, a projection the paper does not make.","The R1-R8 scheme could be reused as a reporting standard for other mobile-manipulation deployments; if multiple teams published the same breakdown, bottleneck comparisons across platforms would become possible, which the paper leaves implicit.","The paper's own diagnosis suggests a concrete experiment it does not run: adding wrist cameras or tactile sensing to track the item through grasp and transport, then measuring whether R3, R5, and R6 failures drop, would directly test the open-loop-execution claim."],"forward_implications":["On this item set, heavy, bulky, or otherwise incompatible items are the single largest failure cause (36 percent overall; 45.6 percent of suction failures), so general grocery picking will require bimanual or alternative nonprehensile strategies.","Open-loop grasp execution is a common thread: failures from displaced items, neighbor interference, and drops during transport (R3, R5, and R6) together account for about 30 percent of misses and would need closed-loop state tracking to address.","A compliant dithering search recovers most initial suction misses (14 of 17), so small compliant probing around the planned pose is an effective and cheap robustness mechanism.","Extraction remains hard in narrow shelf openings even with admittance control; adding rotational compliance during extraction is the paper's stated next step.","The full-stack, in-store test format surfaces bottlenecks such as planning edge cases, kinematic imprecision, and partial observability that tabletop grasping benchmarks abstract away."],"supporting_citations":[{"why":"Supplies the underlying platform, the online full-stack design, and the metrics-driven methodology that this field-test analysis extends.","marker":"[1]"},{"why":"Frames the benchmark challenge of building robotic systems for real picking tasks, the context this work quantifies with 335 attempts.","marker":"[5]"},{"why":"Provides the Robust PCA pose estimation used to determine item orientation for grasp planning.","marker":"[21]"},{"why":"Supplies the object detector that produces bounding boxes for grocery items and feeds the classification and segmentation stages.","marker":"[22]"},{"why":"Provides the antipodal and approach-based grasp sampling schemes on which the grasp pose planners are built.","marker":"[23]"},{"why":"Supplies the robust point-cloud registration method used for runtime tool pose correction.","marker":"[26]"},{"why":"Informs the bimanual nonprehensile strategies explored for the heavy and bulky items that dominate the R1 failure class.","marker":"[27]"}],"fun_headline_variants":["61% success on 335 grocery picks; open-loop execution fails","Eight grasp failure causes from 335 real grocery picks","Real grocery test: 61% success, failures traced to open-loop","335 picks in a real store: eight failure modes identified","Open-loop execution blamed for most grocery grasp failures"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The failure taxonomy is load-bearing, and it rests on the authors' manual labeling of each failed pick into one of eight causes, with no reported labeling protocol or cross-check, so the percentages could shift under different coders.","fun_headline_variants_meta":{"raw":{"variants":["61% success on 335 grocery picks; open-loop execution fails","Eight grasp failure causes from 335 real grocery picks","Real grocery test: 61% success, failures traced to open-loop","335 picks in a real store: eight failure modes identified","Open-loop execution blamed for most grocery grasp failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2861,"prompt_tokens":961,"completion_tokens":1900,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1817}},"tokens_in":577,"tokens_out":1900,"duration_ms":15011,"temperature":1.0,"reasoning_tokens":1817,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:29:08.587385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the recorded failed picks with an independent panel using a written protocol and check whether the R1-R8 distribution is reproduced; if coders disagree or the top cause changes, the reported percentages are not stable ground truth.","supporting_citations":[{"cited_title":"Demonstrating Mobile Ma- nipulation in the Wild: A Metrics-Driven Approach","cited_arxiv_id":null,"evidence_quote":"Supplies the underlying platform, the online full-stack design, and the metrics-driven methodology that this field-test analysis extends."},{"cited_title":"Lessons from the Amazon picking challenge: Four aspects of building robotic systems","cited_arxiv_id":null,"evidence_quote":"Frames the benchmark challenge of building robotic systems for real picking tasks, the context this work quantifies with 335 attempts."},{"cited_title":"Non-convex Robust PCA","cited_arxiv_id":null,"evidence_quote":"Provides the Robust PCA pose estimation used to determine item orientation for grasp planning."},{"cited_title":"End-to-End Object Detection with Transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the object detector that produces bounding boxes for grocery items and feeds the classification and segmentation stages."},{"cited_title":"A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set","cited_arxiv_id":null,"evidence_quote":"Provides the antipodal and approach-based grasp sampling schemes on which the grasp pose planners are built."},{"cited_title":"A Direct Semi-Exhaustive Search Method for Robust, Partial-to-Full Point Cloud Registration","cited_arxiv_id":null,"evidence_quote":"Supplies the robust point-cloud registration method used for runtime tool pose correction."},{"cited_title":"In the Wild Ungras- pable Object Picking with Bimanual Nonprehensile Manipulation","cited_arxiv_id":null,"evidence_quote":"Informs the bimanual nonprehensile strategies explored for the heavy and bulky items that dominate the R1 failure class."}],"review_version":1}