{"id":"13d7481c-51f2-477e-83ec-01169821f937","arxiv_id":"2509.05031","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A transformer with an engineered finger-to-object angle feature matches a geometric baseline (90% accuracy) for pointing target prediction on a small tabletop dataset.","lead":"This paper presents MM-ITF, a transformer model that reads 2D hand poses and object positions from monocular RGB images to predict which object a person is pointing at, scoring 90% accuracy in a tabletop robot study. It is a modest, applied step in human-robot interaction rather than a fundamental advance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Augmentation may leak across the train/test split, invalidating the 90% generalization estimate","rationale":"The reader correctly identified that the validity of the 90% accuracy result depends on the independence of test scenes, but the stated arithmetic inconsistency in Section 4.2 is not actually an inconsistency: 21 train + 3 validation + 6 test = 30 scenes, and 8 folds × 3 validation slots = 24, which leaves exactly 6 held-out test scenes. The more serious and concrete threat is the placement of augmentation in the pipeline. Given that the augmented dataset has over 2.3 million samples derived from 572 original samples via joint geometric transforms, if augmentation precedes the split, the model has effectively seen transformed copies of the test scenes during training—not just same-scene, same-frame data but the same pointing instances at different positions and rotations. This would directly invalidate the central claim of accurate prediction on unseen data. The manuscript also does not report any participant-level split, so even with augmentation after the split, a model could rely on participant-specific pointing habits. These are concrete, checkable issues: the code is publicly linked, so a reviewer can verify the data pipeline and rerun the evaluation. The verdict should remain conditional because the concern is resolvable by inspecting and rerunning the code; the paper should be required to clarify the augmentation-split order and ideally report participant-independent held-out accuracy. The angle-feature concern about the 'no geometric rules' claim is real but secondary: even if the relationship feature is a geometric heuristic, the method still uses monocular RGB and may still be accurate, so it does not undermine the empirical result in the same way a test-set leak would.","tokens_in":9397,"tokens_out":4533,"duration_ms":48440,"concrete_test":"In the released repository (https://github.com/lucamuellercode/MMITF), inspect the data preparation script to determine whether augmentation is applied before the train/test split or per-fold after splitting. Then rerun the three-modality evaluation with augmentation applied only to the 21 training scenes per fold (and no augmented copies of the 6 test scenes in training), ideally with test scenes from participants not seen in training. If the Table 1 accuracy drops materially (e.g., from 90% toward the two-modality 71%), the generalization claim fails; if it remains around 90%, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the three-modality MM-ITF achieves 90% accuracy on held-out scenes (Abstract, Conclusion). Section 3.1 describes a large augmentation pipeline (mirroring, 64 shifts, 8 rotations, 4 noise levels) yielding 2,342,912 samples from 572 source samples, but it does not state whether augmentation is applied before or after the train/validation/test split described in Section 4.2. If augmentation is applied to the full 30-scene dataset before the split, then augmented copies of the six held-out test scenes appear in the 21-scene training sets of every fold. Because the transformations are applied jointly to hand keypoints and object locations, the model could effectively memorize augmented variants of the test scenes, so the reported 90% test accuracy would not measure generalization to unseen scenes. The reader's arithmetic concern about Section 4.2 is a misreading: 21 train + 3 validation + 6 test = 30 per fold, and 8 folds × 3 validation scenes = 24, so the split is internally consistent. The real load-bearing issue is the unspecified placement of augmentation relative to the split, and the absence of participant-level splitting (18 participants, 30 videos), which could also inflate accuracy via identity leakage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MM-ITF, a transformer-based encoder-decoder with inter-modality attention for pointing target estimation in a tabletop HRI scenario. Hand keypoints from MediaPipe and object centroids from OWLv2 are embedded, together with a third input: the angle between the index-finger vector and each object centroid. A non-object token represents resting hands, and the model assigns a score to each candidate object, selecting the highest. The method is evaluated on 30 videos from 18 participants (572 source samples, augmented to 2,342,912), using an eight-fold protocol with 21 training, 3 validation, and 6 held-out test scenes. Reported results: 90% accuracy and 96% top-2 accuracy for the three-modality model, 71% accuracy for the two-modality ablation, and 89% accuracy for the geometric baseline. The paper also introduces a patch confusion matrix for spatial error analysis and claims that the method works from monocular RGB without predefined geometric rules.","tokens_in":9729,"tokens_out":6189,"duration_ms":69746,"significance":"If the evaluation is valid, the paper makes a moderate but useful contribution: it shows that a transformer with inter-modality attention can fuse 2D hand landmarks and object centroids for pointing-target selection, avoiding 3D reconstruction and wearable sensors. The patch confusion matrix is a clear interpretability tool, the code is released, and the two-modality versus three-modality ablation gives insight into the role of the relationship input. However, the central claim that the method works without geometric rules is weakened by the explicit hand-coded angle feature, and the augmentation/split ordering must be clarified before the 90% accuracy figure can be interpreted as a true generalization estimate. Overall, the contribution is appropriate for an applied HRI venue, but the evaluation protocol and wording of the main claim require revision.","major_comments":[{"comment":"The placement of data augmentation relative to the train/validation/test split is not stated. Section 3.1 describes augmentation of the 572 samples to 2,342,912 samples, and Section 4.2 describes the 21/3/6 split, but the text does not say whether augmentation is applied before or after the split. If augmentation is applied before the split, augmented copies of the six held-out test scenes appear in the training folds; since the transformations preserve the pointing direction, the model could memorize test-scene variants and the reported 90% accuracy would not measure generalization to unseen scenes. In addition, the split is at the scene level, not the participant level, and the dataset contains 18 participants with 30 videos; identity leakage could also inflate accuracy. The authors should explicitly state the augmentation/split ordering and provide a participant-disjoint split or repo","section":"§3.1, §4.2"},{"comment":"The abstract and conclusion claim that the method predicts pointing targets 'without relying on predefined geometric rules.' This is not supported as written: the relationship feature θ_i in Eq. (3) is a hand-coded angular alignment between the index-finger vector and each object centroid. The ablation in Table 1 shows that adding this feature raises accuracy from 71% to 90%, so the geometric prior is the main driver of the result. The method avoids geometric post-processing such as line-intersection, but it still relies on a predefined geometric relationship as an input. The authors should reframe the claim as 'without geometric post-processing' or show that the model can learn the angular relationship from pose and object coordinates alone.","section":"§3.2, Eq. (3), §5"},{"comment":"The evaluation protocol is arithmetically consistent if read as a fixed six-scene test set with rotating three-scene validation subsets, but the text calls it 'eight-fold cross-validation,' which is non-standard. More importantly, the reported metrics are averaged over eight models, yet all eight models appear to share the same six test scenes, and neither the number of test samples per scene nor per-fold breakdown is given. With only six test scenes, scene-level correlations can dominate the reported 0.90 accuracy and the ±0.017 standard deviation. The authors should provide per-fold and per-scene results, the number of test frames, and a clear statement of how the six test scenes were selected.","section":"§4.2, Table 1"}],"minor_comments":[{"comment":"The split description should be rewritten for clarity. The phrase 'eight-fold cross-validation' alongside 'trained on 21 scenes and validated on a unique subset of three' and 'six scenes held out as a test set' can be read as inconsistent unless one infers that the six test scenes are fixed. Please state explicitly that the six test scenes are fixed and that the 24 remaining scenes rotate through validation across the eight folds.","section":"§4.2"},{"comment":"The augmentation description says 'eight random shifts along both the x- and y-axes.' This is ambiguous: it could mean eight shifts total, or 8×8 = 64 shifts. The stated final count of 2,342,912 from 572 source samples implies 2 × 8 × 8 × 8 × 4 = 4096-fold augmentation, but the multiplier should be spelled out.","section":"§3.1"},{"comment":"The three-modality model and the baseline differ by only one point (0.90 vs. 0.89), and the reported standard deviations overlap. The paper does not report any significance test or confidence intervals. Since the conclusion emphasizes achieving performance comparable to the baseline, a statistical comparison would strengthen the claim.","section":"Table 1"},{"comment":"The patch coordinates in the confusion matrix (e.g., '(1,12)', '(3,12)') are used in the text but the figure does not define the origin or ordering of the patch grid. Please add axis labels and a description of how patch indices relate to image coordinates.","section":"§4.3, Figure 4"},{"comment":"The title contains a formatting artifact: 'T arget' should be 'Target'.","section":"Title"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the code release is a positive feature. The decisive issue is the unspecified ordering of augmentation and the split: if augmentation was applied only to training folds, the reported 90% accuracy is plausible; if not, the headline result is not a valid generalization estimate. The hand-coded angle feature also needs to be acknowledged as a geometric prior. These are fixable with a revision, so I do not recommend rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a modest but legitimate extension of inter-modality attention to 2D pointing target prediction for a tabletop HRI scenario. The architecture is straightforward, the comparison against Ali et al.'s 2D baseline is fair, and the patch confusion matrix is a reasonable interpretability tool. The three-modality model reaches 90% accuracy, one point above baseline, and the two-modality version at 71% shows the angle feature is doing much of the work.\n\nWhat's genuinely new: applying transformer inter-modality attention (from HOI detection) to deictic pointing in a fixed camera/table setup, plus the patch-level confusion matrix. The authors are transparent about the setup's limitations—fixed camera, table, ten known YCB objects, 18 participants—and they don't oversell the architecture's generality.\n\nThe soft spots are real, though. The central claim in the abstract and conclusion—that the method works 'without relying on predefined geometric rules'—doesn't survive contact with Section 3.2. Equation 3 is exactly a hand-coded geometric rule: the angle between the index finger vector and the vector to each object centroid. The model learns to use that feature, and the 19-point jump from two-modality to three-modality accuracy shows it's the decisive input. So the framing is misleading even if the empirical comparison is honest.\n\nSecond, the data split needs a clear statement. The eight-fold description (21 train, 3 validation, 6 held-out test per fold) is internally consistent, contrary to the reader's arithmetic concern. But the paper never says whether the augmentation pipeline described in Section 3.1 is applied before or after the split. If augmented copies of the six test scenes end up in the 21-scene training sets, the reported test accuracy measures memorization of transformed test scenes, not generalization. The augmentation section says transformations are applied to hand keypoints and object locations jointly, so that's a plausible leak. The authors also don't split by participant; 30 videos from 18 people can leak identity. These are fixable with a one-paragraph clarification, but they're load-bearing for the generalization estimate.\n\nFinally, reproducibility is thin: no hyperparameters for the transformer (d_T, layers, heads, optimizer, epochs, batch size), no augmentation schedule, and the code link is given but not verified here. For a 2.3M-sample augmentation from 572 source samples, the exact split placement and random seed matter.\n\nOverall: the paper deserves peer review. The empirical setup is controlled and the comparison is clean enough, but the 'no geometric rules' claim should be softened or removed, and the split/augmentation protocol needs to be explicit. A serious referee would ask for those clarifications before acceptance. It's a reasonable contribution for an HRI workshop or short conference paper, but not a high-impact result.","headline":"A competent tabletop pointing paper whose empirical claim is plausible, but whose 'no geometric rules' framing is undercut by its own hand-coded angle feature and whose data split needs clarification before the 90% number is taken at face value.","tokens_in":10177,"tokens_out":2877,"would_cite":false,"duration_ms":29123,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer with inter-modality attention can map monocular RGB pointing gestures to the intended tabletop object with 90% accuracy, matching a geometric baseline without extra hardware.","keywords":["pointing gesture recognition","human-robot interaction","transformer attention","inter-modality fusion","object target estimation","deictic gesture","monocular RGB","tabletop robotics"],"falsifier":"Re-run the eight-fold experiment with strict participant-disjoint splits, and additionally separate the augmented samples by their source video; if top-1 accuracy on never-seen participants drops well below 90%, the result was driven by scene/participant correlations rather than a general pointing-to-object mapping. A second check: collect trials where two objects lie on the same pointing ray and measure accuracy on those alone, since the paper's own patch matrix reports the model often selecting the farther object.","tokens_in":9349,"feed_emoji":"👉","tokens_out":5428,"duration_ms":59883,"temperature":0.7,"pith_summary":"MM-ITF is an attempt to make pointing-target prediction from a single RGB camera work by learning, rather than geometrically constructing, the link between a hand and the object it indicates. The model feeds 21 hand landmarks, object centroids, and a hand-coded finger-to-object angle into a transformer encoder-decoder, with hand tokens attending to object tokens, and outputs a likelihood score for every candidate object plus a non-object class. On a controlled tabletop task with a social robot, the method reports 90% top-1 and 96% top-2 accuracy, matching a line-projection baseline while predicting pointing state and target in one forward pass. If the result holds, it means a robot can interpret human deictic intent from cheap monocular RGB data alone, without 3D reconstruction, calibration, or wearables.","feed_headline":"RGB-only transformer reads pointing gestures at 90% accuracy","feed_subtitle":"Hand landmarks and object centroids attend to each other, letting a robot rank targets without 3D sensors.","key_machinery":"The load-bearing mechanism is the inter-modality attention block: hand-pose tokens act as queries against object-location keys and values, so each hand landmark aggregates object context, forming a pose-object memory. A separate relationship token per hand-object pair—the cosine angle between the index-finger vector and the centroid vector, plus a non-relation token of -1 for non-pointing hands—is fed to the decoder, which self-attends and then cross-attends to the encoder memory; a feedforward network with sigmoid assigns each object a score, and the highest score is the prediction. The non-object token (-1,-1) gives the same architecture an explicit 'not pointing' output.","core_discovery":"The paper's central claim is that a pointing gesture can be treated as a relation between hand and object tokens and scored by inter-modality attention: the encoder lets hand pose features attend to object locations, the decoder lets relationship tokens—the angle between the index finger and each object centroid—attend to that fused memory, and a sigmoid output ranks the objects. With all three modalities the architecture reaches 90% accuracy and 96% top-2 accuracy, slightly above the 89% of the 2D geometric baseline. This is presented as evidence that the intended object can be predicted accurately using monocular RGB data, and that the hand-coded angular relationship is what lifts the mode","pith_inferences":["The 19-point jump when adding the angle suggests the attention mechanism itself may be learning proximity and alignment, while the crisp selection is effectively delegated to the hand-coded cosine angle; a test that removes only the angle and replaces it with a learned direction embedding would separate those contributions.","From the patch confusion matrix patterns, the model often picks an object farther along the pointing ray behind the true target; a targeted evaluation on collinear-object configurations would show whether this is a systematic bias.","The same scorer could be trained for gaze direction, head pose, or other deictic cues by changing only the query tokens, and the non-object token gives a natural 'no referent' output for those modalities.","A strict participant-disjoint and scene-disjoint evaluation would be the natural next check that the accuracy is a statement about reading novel pointing gestures rather than about recognizing familiar layouts."],"forward_implications":["A robot can rank the likely pointing targets in a single forward pass, so a downstream planner can ask for confirmation or grasp the most likely one.","Monocular RGB is sufficient in the controlled tabletop regime, removing the need for depth sensors, IMUs, or multi-camera calibration.","The two-modality result (71%) shows that pose and location alone learn coarse relations (top-2 92%); the angular relationship is what refines the final choice, which means explicit relational geometry still matters inside a learned pipeline.","The patch confusion matrix offers a way to inspect where spatial errors concentrate—such as objects aligned along the pointing ray—rather than aggregating over arbitrary object indices.","The architecture is modular: gaze or other cues could be added as additional tokens without changing the scoring head."],"supporting_citations":[{"why":"Supplies the inter-modality attention mechanism that lets hand-pose tokens attend to object tokens.","marker":"[13]"},{"why":"Provides the 21 hand landmarks per hand used as the pose input tokens.","marker":"[30]"},{"why":"Provides the open-vocabulary object detector whose bounding-box centroids form the object tokens.","marker":"[22]"},{"why":"Baseline method: an MLP plus proximity to the wrist-to-index line, applied in the same NICOL tabletop scenario.","marker":"[1]"},{"why":"Source of the transformer encoder-decoder design and sinusoidal positional encoding the model adapts.","marker":"[8]"},{"why":"Defines the ten standard YCB objects placed on the table in the evaluation dataset.","marker":"[6]"},{"why":"Describes the NICOL robot platform whose embedded camera captured the dataset.","marker":"[14]"}],"fun_headline_variants":["Transformer maps pointing gestures to objects with 90% accuracy","Pointing at robots: RGB-only transformer hits 90% target accuracy","Inter-modality attention lets robots read pointing via RGB alone","90% accuracy: transformer decodes pointing for robot intent","No depth needed: transformer ranks objects from pointing and RGB"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported accuracy assumes the held-out test scenes—and the participants and augmented copies associated with them—are truly unseen in training; if that separation leaks, 90 percent overstates how well the model reads new pointing gestures.","fun_headline_variants_meta":{"raw":{"variants":["Transformer maps pointing gestures to objects with 90% accuracy","Pointing at robots: RGB-only transformer hits 90% target accuracy","Inter-modality attention lets robots read pointing via RGB alone","90% accuracy: transformer decodes pointing for robot intent","No depth needed: transformer ranks objects from pointing and RGB"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2116,"prompt_tokens":701,"completion_tokens":1415,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":1331}},"tokens_in":445,"tokens_out":1415,"duration_ms":11079,"temperature":1.0,"reasoning_tokens":1331,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:39:17.604331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the eight-fold experiment with strict participant-disjoint splits, and additionally separate the augmented samples by their source video; if top-1 accuracy on never-seen participants drops well below 90%, the result was driven by scene/participant correlations rather than a general pointing-to-object mapping. A second check: collect trials where two objects lie on the same pointing ray and measure accuracy on those alone, since the paper's own patch matrix reports the model often selecting the farther object.","supporting_citations":[{"cited_title":"In: ICCV","cited_arxiv_id":null,"evidence_quote":"Supplies the inter-modality attention mechanism that lets hand-pose tokens attend to object tokens."},{"cited_title":"In: Proceedings of the 37th International Conference on Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Provides the open-vocabulary object detector whose bounding-box centroids form the object tokens."},{"cited_title":"ICSR (2024)","cited_arxiv_id":null,"evidence_quote":"Baseline method: an MLP plus proximity to the wrist-to-index line, applied in the same NICOL tabletop scenario."},{"cited_title":"In: Vedaldi, A","cited_arxiv_id":null,"evidence_quote":"Source of the transformer encoder-decoder design and sinusoidal positional encoding the model adapts."},{"cited_title":"In: International Conference on Advanced Robotics ICAR","cited_arxiv_id":null,"evidence_quote":"Defines the ten standard YCB objects placed on the table in the evaluation dataset."},{"cited_title":"IEEE Access 11 (2023)","cited_arxiv_id":null,"evidence_quote":"Describes the NICOL robot platform whose embedded camera captured the dataset."}],"review_version":1}