{"id":"b4751768-0d69-4a01-b0b5-fda8f3372ce8","arxiv_id":"2607.10706","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Projecting 3D gripper keypoints onto camera pixels and classifying those pixels yields millimeter-precise, multi-modal closed-loop manipulation faster than diffusion policies.","lead":"Action Map Policy turns robot arm control into predicting which image pixels the gripper will pass through, then lifts those pixels back to 3D poses. It beats diffusion and regression baselines on simulated and real manipulation while running in one network pass.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged triangulation/calibration dependence.","rationale":"The central claim is an empirical methods claim: representing closed-loop 3D actions as multi-view keypoint heatmaps and training with cross-entropy yields higher success, faster single-pass inference, and better fine-grained spatial response than regression/token/diffusion baselines, with ~1 mm theoretical precision at 224×224. The paper supplies consistent evidence (Tables 2–5, real-world completion rates, inference timings, laser-pointer tasks) and a clear precision argument (Table 1 / Fig. 4) that conventional per-dimension discretization would need ~10^10 tokens for comparable accuracy. No load-bearing mathematical inconsistency appears; the bijection claim is carefully scoped to generic cameras and Im(P). The softest link is exactly the one the reader named—recovery of executable 3D trajectories from independent heatmap argmaxes under real geometry—not a hidden flaw in the classification formulation or the X-Net design. Evaluation hygiene (best-checkpoint, no error bars) and missing public artifacts are real but secondary to correctness of the claim under the stated assumptions. Therefore the reader's CONDITIONAL verdict (accept-shaped if checkpoint selection, uncertainty, and code/data release are addressed) should stand; no stronger rejection or unconditional acceptance is warranted from a second-pass stress test.","tokens_in":15390,"tokens_out":766,"duration_ms":9899,"concrete_test":"On the real coffee/toast/egg logs (or a held-out replay set), recompute 3D keypoints two ways for every executed chunk: (i) full multi-view triangulation of network argmaxes, and (ii) single-view depth-lifted argmax when one side view is masked or when a keypoint is near the image border. Report mean translation/rotation error of recovered poses and task completion rate under forced single-view or ±5 mm / ±2° calibration perturbation. If completion drops >20 points or pose error exceeds ~3 mm / ~3°, the closed-loop claim weakens outside ideal calibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption correctly isolates the most load-bearing condition: that argmax over independently predicted multi-view heatmaps, followed by triangulation with known camera matrices (Method §3; Algorithms 1–2), recovers accurate, temporally consistent 3D keypoint trajectories under real calibration error, occlusion, and out-of-view motion. The paper itself notes the workspace is limited by calibrated coverage (§5) and that imperfect calibration introduces small offsets that the model must absorb during training (footnote 2 / §4.1). Table 1 and Figure 4 quantify only ideal reconstruction error of the discretized (P,T) map on real-world data, not closed-loop policy error under the full pipeline (soft labels → network → argmax → triangulation → pose recovery). Soft labels and equivariant augmentation help training, but do not guarantee that independent per-view argmaxes remain consistent enough for Gram–Schmidt pose recovery when one view is occluded or a keypoint leaves the FOV (truncation rule in Appendix 6.2). This is a real engineering dependence, not an internal contradiction; the reported sim/real gains and laser-pointer spatial-reasoning results still support the central claim under the paper's calibrated multi-view setup.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Action Map Policy (AMP), which reformulates 3D closed-loop visuomotor policy learning as multi-view pixel classification: end-effector poses are encoded as a fixed set of 3D keypoints (m=5), projected into calibrated camera planes, and supervised as soft Gaussian heatmaps over a temporal action chunk. An X-Net backbone (U-Net encoders, multi-view transformer with in- and cross-image attention, U-Net decoders) predicts dense heatmaps; inference recovers 3D keypoints by per-view argmax plus triangulation, then reconstructs pose via centroid and Gram–Schmidt axes. The method is trained with cross-entropy only, yields single-pass action chunks, and is evaluated on six MimicGen tasks plus five real-world tasks (three long-horizon, two laser-pointer spatial-reasoning), reporting higher success rates than DiffPo, ACT, OAT, and Motion Track, faster inference than diffusion, and millimeter-scale reconstruction precision at 224×224 (Table 1 / Fig. 4).","tokens_in":15765,"tokens_out":1360,"duration_ms":19218,"significance":"If the empirical gains hold under fair comparison, AMP is a useful contribution to action representation for imitation learning: it sidesteps combinatorial token vocabularies while retaining multi-modality via classification, aligns action and observation in image space (enabling equivariant augmentation), and offers single-pass inference that is practically faster than iterative diffusion. The precision analysis (Table 1) is concrete and falsifiable; the laser-pointer experiments give a clear demonstration of fine-grained spatial responsiveness that regression and diffusion baselines struggle with. Strengths include a clean geometric pipeline (projection/triangulation, Algorithms 1–2), ablations on soft labels and in-hand view (Table 3), and real-robot validation on long-horizon tasks with reported wall-clock latency. These make the work of interest to the robot learning community even if some engineering dependencies remain.","major_comments":[{"comment":"Table 2 (and Tables 4–5): success rates are reported as single percentages over 50 (sim) or 20 (real) trials with no standard errors, confidence intervals, or multi-seed statistics. The headline average gain of ~20.7% over DiffPo and the large real-world margins (e.g., 80–90% vs 15–40%) are load-bearing for the central claim; without variance or seed-level reporting it is hard to judge whether differences are robust, especially on threading-d2 (30% vs 26%) and under the small real-world demo budgets (70/40 demos).","section":null},{"comment":"Method §3 and Algorithms 1–2: inference treats each view’s heatmap independently (argmax then triangulation T). Table 1 / Fig. 4 quantify only ideal discretized (P,T) reconstruction error on ground-truth projections, not closed-loop error under network prediction, occlusion, or partial FOV loss. The paper notes workspace limits and calibration offsets (§5, footnote 2) and truncates out-of-scope keypoints (Appendix 6.2), but does not measure cross-view argmax consistency or pose recovery failure rates when one view is occluded. A quantitative consistency/ablation (e.g., drop one camera at test time; report triangulation residual and success) is needed to support the claim that millimeter-level policy precision is achieved in the full pipeline, not only in the geometric map.","section":null},{"comment":"§4.2 baselines and Appendix 6.4: equivariant joint image–heatmap augmentation is a structural advantage of AMP (Table 7 shows large drops without it: −32/−26/−12 points). It is unclear whether DiffPo, ACT, OAT, and Motion Track received comparably strong multi-view geometric augmentation or only standard image augmentations. If baselines were not given an analogous geometric prior, part of the reported gap may be attributable to data augmentation rather than the classification action map itself. Please state the exact augmentation protocol for every baseline and, if possible, re-run the strongest baseline with the strongest applicable multi-view augmentation.","section":null}],"minor_comments":[{"comment":"Figure 1 caption and §1: the comparison of multi-modality handling is clear conceptually, but a short quantitative multi-modality diagnostic (e.g., entropy of heatmaps vs mode collapse rate of ACT on the laser-pointer tasks) would strengthen the narrative beyond success rates alone.","section":null},{"comment":"§3, gripper width formula: w = d+/(d++d−) with p5 is fine for a parallel jaw, but the text should state explicitly that p1–p4 distances are fixed and cannot encode aperture (currently only in footnote 1).","section":null},{"comment":"Implementation: decoder outputs 60 channels (5 keypoints × 12 steps) but only the first 8 steps are executed; justify the train/execute horizon mismatch and whether longer executed horizons hurt consistency.","section":null},{"comment":"Related work: Motion Track [39] and dense open-loop methods [17–22] are discussed; a clearer sentence on how AMP differs from 2D keypoint diffusion (independent coordinate denoising vs joint dense classification with cross-view attention) would help readers place the contribution.","section":null},{"comment":"Typos / polish: abstract and intro repeat similar claims; “Muti-view” in Fig. 2; arXiv IDs in references for concurrent work (OAT, etc.) should be double-checked for final citation form.","section":null},{"comment":"Appendix 6.5 visualization is helpful; consider moving one qualitative multi-view heatmap figure into the main paper near the spatial-reasoning results.","section":null}],"recommendation":"minor_revision","confidential_remarks":"Central idea is solid and the real-robot spatial-reasoning results are among the more convincing parts. The main risk is over-attributing gains to the action representation when equivariant augmentation and multi-view architecture also differ from baselines. I would not reject on the triangulation dependence alone—the paper is honest about FOV limits—but statistical reporting and baseline augmentation parity should be fixed before acceptance. Scope fits a robotics / robot-learning venue well."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a solid robotics methods paper. Casting closed-loop 3D action chunks as multi-view pixel classification of projected gripper keypoints is a coherent, usable formulation, not just a rebrand of Transporter or Motion Track.\n\nWhat is new is the closed-loop package: m=5 keypoints over a horizon as soft heatmaps, X-Net (U-Net + multi-view transformer) that keeps dense spatial structure, cross-entropy training, single-pass argmax + triangulation to pose. The precision table is useful—~1 mm / 1.3° at 224² without a 10^10 token codebook—and the laser-pointer tasks actually show the spatial-reasoning claim (100% vs DiffPo ~20%). Sim gains on six MimicGen tasks and real coffee/toast/egg completion rates are large and consistent with the story. Inference is genuinely faster than DDIM DiffPo. Equivariant image–heatmap augmentation is a natural fit and the ablations (in-hand, σ, aug) are honest.\n\nSoft spots, in proportion: they report best-checkpoint success without error bars or fixed-seed protocol; that is sloppy but not unique in this literature. The load-bearing engineering assumption is real—independent per-view argmax then triangulation under calibration error, occlusion, and FOV limits (they note workspace coverage and absorb small offsets in training). Table 1 is ideal reconstruction, not full closed-loop error under the pipeline. Hand-designed keypoints and known extrinsics are part of the setup, not hidden. No public code/data yet. None of that overturns the reported gains under their multi-camera regime.\n\nWho it is for: people building multi-camera visuomotor policies who care about multi-modality, speed, and pixel-level cues. Math is elementary geometry + CE; citations cover the right baselines. I would bring it to reading group, cite it when discussing action representations, and send it to referees. Fix checkpoint selection, add uncertainty, release artifacts, and it is a clear accept-shaped contribution.","headline":"Clean closed-loop idea: multi-view keypoint heatmaps + CE + triangulation beats DiffPo/ACT/OAT on MimicGen and real long-horizon tasks, with real speed and laser-cue wins; main soft spots are eval hygiene and calibrated multi-view dependence, not a broken claim.","tokens_in":16366,"tokens_out":539,"would_cite":true,"duration_ms":5963,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"3D robot actions can be learned as pixel classification on camera images, not as continuous regression or huge discrete tokens.","keywords":["robot manipulation","imitation learning","action representation","pixel classification","keypoint heatmaps","closed-loop visuomotor policy","multi-view triangulation","diffusion policy baseline"],"falsifier":"On the real coffee, toast, or egg tasks, replace the argmax-plus-triangulation step with ground-truth projected pixels or with deliberately noisy heatmaps; if success rates collapse or fail to match the reported gains over diffusion and ACT, the claim that pixel classification plus triangulation is sufficient fails.","tokens_in":16302,"feed_emoji":"🤖","tokens_out":842,"duration_ms":8333,"temperature":0.7,"pith_summary":"Robot policies struggle with high-dimensional, multi-modal, long-horizon actions. This paper claims that the right fix is to stop treating actions as free continuous vectors or as an exploding codebook of tokens. Instead it projects gripper keypoints into the camera image planes and asks the network to classify which pixel each keypoint should hit at each future step. The resulting heatmaps keep multi-modality, preserve millimeter-scale precision at ordinary image resolutions, and let the whole action chunk be predicted in one forward pass. On simulated and real manipulation tasks the method beats strong regression, token, and diffusion baselines on success rate, inference speed, and response to tiny spatial cues such as a laser pointer. A sympathetic reader cares because the formulation aligns observation and action in the same spatial grid, turning policy learning into ordinary dense classification while still recovering executable 3D poses by triangulation.","feed_headline":"Robot actions learned as pixel maps beat diffusion on speed and success","feed_subtitle":"Project gripper keypoints into camera images; classify pixels; triangulate back to 3D poses.","key_machinery":"Action Map Policy (AMP) with the X-Net backbone: 3D end-effector poses are converted to a fixed set of gripper keypoints, projected into multi-view soft heatmaps, and the network is trained with cross-entropy to predict those heatmaps over a temporal horizon; executable actions are recovered only at inference by argmax plus triangulation.","core_discovery":"Action Map Policy shows that 3D closed-loop manipulation can be cast as multi-view pixel classification of projected action keypoints. Predicting dense heatmaps over image planes, then recovering poses by argmax and triangulation, yields higher success rates than regression, tokenization, and diffusion baselines, single-pass inference much faster than diffusion, and stronger use of fine-grained visual cues, all without a combinatorial action vocabulary.","pith_inferences":["The classification objective may transfer more cleanly into vision-language-action models than continuous regression or diffusion, because both language and pixel heatmaps are already discrete token spaces.","Independent per-view argmax can still produce inconsistent 3D geometry under heavy occlusion; a joint multi-view consistency regularizer would be a natural next test.","If the heatmaps are already multi-modal, sampling from them rather than always taking argmax could give a cheap way to explore alternative grasps without a second generative model."],"forward_implications":["Millimeter translation and degree-level rotation precision become available at ordinary 224\times224 resolution without building a 10^10-token action codebook.","Closed-loop action chunks can be generated in a single forward pass, removing the need for multi-step denoising schedules at inference.","Because observation and action live on the same pixel grid, equivariant image augmentations transfer directly to the action labels.","The explicit per-pixel action distribution supplies a natural interface for later reweighting or reinforcement-learning fine-tuning.","The same keypoint-heatmap design extends to other end-effectors by changing only the geometric keypoint layout."],"fun_headline_variants":["Pixel classification maps 3D robot actions, beating diffusion speed and success","AMP: project keypoints to pixels, classify heatmaps, triangulate 3D poses","Single-pass pixel maps learn closed-loop 3D manipulation faster than diffusion","Action maps cast robot policy as multi-view pixel classes for higher success","Classify projected gripper pixels then triangulate: AMP tops regression baselines"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That the single brightest pixel in each independently predicted multi-view heatmap, once triangulated with the known cameras, recovers accurate and temporally consistent 3D keypoint trajectories under real calibration error, occlusion, and motion that leaves the calibrated workspace.","fun_headline_variants_meta":{"raw":{"variants":["Pixel classification maps 3D robot actions, beating diffusion speed and success","AMP: project keypoints to pixels, classify heatmaps, triangulate 3D poses","Single-pass pixel maps learn closed-loop 3D manipulation faster than diffusion","Action maps cast robot policy as multi-view pixel classes for higher success","Classify projected gripper pixels then triangulate: AMP tops regression baselines"]},"model":"grok-4.5","effort":"low","cost_usd":0.00377,"raw_usage":{"total_tokens":1175,"prompt_tokens":772,"num_sources_used":0,"completion_tokens":82,"cost_in_usd_ticks":37700000,"prompt_tokens_details":{"text_tokens":772,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":321,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":772,"tokens_out":82,"duration_ms":4488,"temperature":1.0,"reasoning_tokens":321,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T09:51:33.920162+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the real coffee, toast, or egg tasks, replace the argmax-plus-triangulation step with ground-truth projected pixels or with deliberately noisy heatmaps; if success rates collapse or fail to match the reported gains over diffusion and ACT, the claim that pixel classification plus triangulation is sufficient fails.","supporting_citations":[],"review_version":1}