{"id":"a9ff6958-417c-4151-bb1c-77b422a5ecf3","arxiv_id":"2507.00446","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"A model-free algorithm estimates the image Jacobian for every pixel in real time and uses it for robot self-body labeling and visual servoing, shown on a musculoskeletal humanoid.","lead":"This paper presents DIJE, an algorithm that estimates a per-pixel image Jacobian in real time using optical flow and a simplified Kalman filter. It then uses that dense map to label the robot's own body and to drive a markerless visual servoing controller, demonstrated on the Musashi humanoid.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The warp-update in Eq. (14) is a bootstrapped transport of the Jacobian estimate; without a stability or convergence analysis, the estimator may settle on a self-consistent but incorrect map, which would undermine both claimed applications.","rationale":"Both the self-recognition and visual servoing claims depend on the dense image Jacobian having a consistent relationship to the robot's body over time. Eq. (14) is the only mechanism that propagates Jacobian information across time and across pixels for a moving body, and it does so using the current estimate of that same Jacobian. This creates a closed loop: the estimated J determines where prior J values are transported, and the KF update then uses observed flow to correct whatever arrives at each pixel. If the initial estimate is zero, the warp initially does nothing; subsequently the estimate must be good enough for the warp to track the correct body point. No argument is given for why this bootstrap succeeds, and the experiments, while encouraging, are insufficient to rule out convergence to a self-consistent but wrong map. The paper is honest about other limitations (static body parts are not recognized; no depth; no occlusion handling), but the bootstrap issue is more central because it threatens both proposed applications. The reader's weakest assumption captures this concern exactly, so no verdict change is needed. Rather, the conditional acceptance should hinge on the synthetic ground-truth test described above, which would disambiguate correct Jacobian estimation from self-consistent but incorrect estimation.","tokens_in":10209,"tokens_out":5443,"duration_ms":69752,"concrete_test":"Simulate a 2-DOF planar arm with known kinematics rendered as a smooth-shaded object, with a static camera. Compute ground-truth dense image Jacobian analytically from the rendering. Run DIJE exactly as described (zero initialization, same KF parameters) on synthetic optical flow plus known joint velocities, and compare the estimated J^D on the arm pixels to ground truth after each timestep. If the estimation error does not decrease to near zero in regions with adequate excitation, or if the estimate settles at a fixed point that predicts observed flow but differs from ground truth in direction, the concern lands. Also run a perturbation check: re-initialize J after convergence with small random values and see whether the estimate returns to the same fixed point; if it does not, the bootstrap has multiple attractors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the warp-update in Eq. (14) preserves and improves the dense image Jacobian estimate when the estimate itself is wrong. Starting from J=0, each timestep the algorithm moves the previous Jacobian value at pixel i to the location predicted by its own current estimate, J_{k-1,i} \\dot q, and then applies the Kalman correction. This is a bootstrapped transport of the state estimate: errors in J affect where the state is transported, and the observation correction acts on the transported state. The paper provides no convergence or stability analysis for this coupled process, and the experiments are qualitative single-run demos on one musculoskeletal robot. It is plausible that the estimate converges to a field that reproduces the observed optical flow under the applied joint motions (so the residual in Eq. (10) is small) yet is not the true image Jacobian for any body point, because the transport has systematically misassociated Jacobians with image locations. Visual servoing may still appear to work if the closed loop only needs a signed gradient direction, and self-recognition may still label the body if the wrong Jacobian is temporally consistent; neither experiment would detect the discrepancy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DIJE, a dense per-pixel image Jacobian estimator that combines dense optical flow with a simplified Kalman filter and a frame-to-frame warp update. The same estimated Jacobian field is then used for two applications: a k-means-based self-recognition scheme that labels the robot's body while rejecting background motion, and a markerless visual servoing controller that drives a user-selected self-body point to a target image point. Experiments on the musculoskeletal humanoid Musashi demonstrate reaching, rake-tip, and vacuum-tool-tip control, together with a qualitative comparison against an observed-flow warp update for self-recognition. The central claim is that a single model-free, real-time dense visuomotor representation can support both body awareness and control without kinematic knowledge or markers.","tokens_in":10588,"tokens_out":5684,"duration_ms":70737,"significance":"If the method is sound, the contribution is significant for robot learning and uncalibrated visual servoing: a dense, real-time image Jacobian estimated from raw observations without markers or a kinematic model, and used for both perception and control, is an appealing and potentially general framework. The paper's strengths include a clear problem formulation, a real-time implementation on a physical musculoskeletal robot, and honest discussion of acknowledged limitations such as the failure to label static body parts and the absence of depth. However, the significance is tempered by the lack of a derivation for the Kalman filter equations, the absence of convergence or stability analysis for the bootstrapped warp update, and the purely qualitative single-run experimental evaluation; these issues need to be addressed before the paper's claims can be regarded as fully supported.","major_comments":[{"comment":"The observation matrix Mq is defined as [qdot^T; qdot^T], which is 2 x N_j, but it multiplies the state vector j of length 2 N_j. This is a dimensional mismatch: the correct form should be a block-diagonal matrix diag(qdot^T, qdot^T) of size 2 x 2 N_j. As written, the observation model u = Mq j is inconsistent with Eq. (10)-(11), and readers cannot reproduce the estimator from the paper. Please correct this equation and provide the surrounding derivation.","section":"III-A, Eq. (3)"},{"comment":"The simplified Kalman filter equations are stated without derivation, and the claim that the covariance symmetry in Eq. (6) is preserved 'by mathematical induction' is not proven. Since these equations are the core of the proposed estimator, the full derivation, including the diagonal-covariance approximation and the equal-x/y-variance reduction, should be included in an appendix or supplementary material. Without this, the approximations cannot be checked and the relationship to the standard Kalman filter remains opaque.","section":"III-A, Eqs. (8)-(11)"},{"comment":"The warp update J_{k|k-1,i} <- interpolate(J^D_{k-1}, x_i - J_{k-1,i} qdot) is a bootstrapped transport: the current Jacobian estimate determines the location to which the previous estimate is warped. Since J is initialized to zero, the predicted flow is initially zero and the warp is identity, and if the estimate is biased, the bias can be transported to incorrect image locations in a self-consistent way. The paper provides no convergence or stability analysis for this coupled estimator, and the experiments do not detect such a failure because both servoing and self-recognition can succeed with a temporally consistent but incorrect Jacobian field. Please provide a synthetic or ground-truth validation (e.g., comparing the estimated dense Jacobian to one computed from a known kinematic and camera model) and, ideally, an analysis or empirical study of error growth under the warp update.","section":"III-B, Eq. (14)"},{"comment":"All experiments are single-run qualitative demonstrations without error bars, baselines, or quantitative performance metrics. The abstract and introduction claim robustness and verifiability, but the reported evidence is not sufficient to support these claims. Please add repeated trials with quantitative metrics: labeling accuracy or IoU against manually labeled ground truth for self-recognition, steady-state error and convergence time for visual servoing, and comparisons against the observed-flow update in Eq. (13) as well as a marker-based or kinematic-model baseline.","section":"V, Experiments"},{"comment":"The paper acknowledges that the self-recognition method does not label non-moving parts of the robot, citing the static shoulder in Fig. 2. This is a substantive limitation of the central 'self-recognition' claim: the output is not a dense binary label of the robot's body, but only of the body parts that are currently moving in a way correlated with the robot's joints. The contribution statement and abstract should be revised to reflect this scope, or the method should be extended to identify static self-parts.","section":"VI, Discussion"}],"minor_comments":[{"comment":"The feedback gain is stated as 'r = 0.034' in the reaching experiment, but r is already used for the observation noise variance in the Kalman filter; this is likely a typo for kp.","section":"V-B"},{"comment":"The text says 'ptarget converges to the target point ptarget' in the rake and vacuum experiments; this should be 'pself converges to ptarget.'","section":"V-B"},{"comment":"There is a typo, 'cosider' should be 'consider.'","section":"VII, Conclusions"},{"comment":"The consistency update formula and the normalization of EvalList are introduced without justification, and the values N_kmeans=5 and e_thresh=0.2 appear to be chosen heuristically; a sensitivity analysis or a brief rationale would improve reproducibility.","section":"Algorithm 1"},{"comment":"The labels in Fig. 4, especially 'External movement,' could be confused with the ground-truth label; please clarify which regions correspond to the robot and which to the background in the figure caption.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and I found no evidence of problematic citation practices. The main issue is that the current version reads as a promising system demonstration rather than a fully validated method; if the authors can supply the missing derivation, a stability or ground-truth check for the warp update, and a quantitative multi-trial evaluation, the contribution would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real systems contribution. The dense per-pixel image Jacobian, the diagonal-covariance Kalman simplification, and the predicted-flow warp (Eq. 14) are new relative to the sparse Jacobian literature, and the hardware experiments on the musculoskeletal robot are believable. The method runs at 30fps and demonstrably handles background motion and tool-tip servoing without markers or kinematic models.\n\nWhat I credit: the diagonal covariance idea is sensible and actually needed for dense scaling; the predicted-flow warp is a clever fix for the label-leak problem that occurs when you warp with observed optical flow, and the side-by-side in Fig. 4 makes the point. The self-recognition via k-means on Jacobian clusters plus temporal consistency is heuristic but works, and the authors are upfront that static body parts are not segmented.\n\nSoft spots, in proportion: the dimensional error in Eq. (3) is real: Mq as written is 2×Nj, which cannot multiply the 2Nj state j. The Kalman update equations (10)-(11) are stated without derivation; the claim that the diagonal covariance is preserved by induction is plausible but not shown. The experiments are single-run, with no error bars, baselines, or ablation of the KF parameters. That matters, but for a systems paper with a working hardware demo, it is a validation gap rather than a fatal flaw.\n\nThe deeper concern is the bootstrapped warp: Eq. (14) transports the Jacobian estimate using flow predicted by that same estimate. If the initial estimate is wrong, the warp can systematically misassociate Jacobians with image locations, and the estimate could settle into a self-consistent but incorrect map. The paper gives no analysis of this, and the closed-loop experiments would not necessarily reveal it, since servoing only needs a gradient-like direction. I don't think this is lethal—many online estimation methods lack formal stability proofs—but it is the one place where I'd want more than plausibility.\n\nBottom line: this paper deserves a serious referee. It's a clear, honest, novel-enough systems contribution whose limitations are openly acknowledged. I'd send it to review with a request to fix the typo, sketch the KF derivation, and either add repeated runs with statistics or label the plots as illustrative.","headline":"A dense per-pixel image Jacobian estimator with a self-warp update that is clever and works on hardware, but the paper lacks derivations, baselines, and any analysis of the bootstrapped warp.","tokens_in":11040,"tokens_out":3211,"would_cite":true,"duration_ms":34892,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a dense per-pixel image Jacobian, estimated online from optical flow and joint velocity by a simplified Kalman filter, lets a robot with no kinematic model and no markers recognize its own body and visually servo to…","keywords":["dense image Jacobian","visual servoing","self-recognition","Kalman filter","optical flow","musculoskeletal robot","model-free control","markerless"],"falsifier":"Repeatedly run the visual servoing experiment with the dense Jacobian initialized to different nonzero matrices (rather than zeros) and record the joint angles at convergence; if the robot consistently reaches the clicked target in image space but ends up in measurably different arm postures across initializations, the warped update has frozen a self-consistent but incorrect Jacobian rather than converging to the true kinematic map.","tokens_in":10055,"feed_emoji":"🤖","tokens_out":7523,"duration_ms":71281,"temperature":0.7,"pith_summary":"This paper proposes DIJE, an algorithm that learns an image Jacobian for every pixel of the camera image, giving a robot a dense map of how each of its joints moves each point on its body. The map is estimated online from optical flow and joint velocity using a simplified Kalman filter, without markers and without any prior knowledge of the robot's kinematic structure. With this map alone, the robot can segment its own body from the background even when external motion overlaps, and can control an unmarked point on its body or on a held tool to track a target in the image. Experiments on a musculoskeletal humanoid show reaching, rake-tip, and vacuum-tip control, all starting from a zero-initialized Jacobian. The paper's broader claim is that a dense visuomotor policy of this sort can unify self-perception and control.","feed_headline":"Robot learns a per-pixel body map with no model","feed_subtitle":"DIJE estimates every pixel's response to joint motion, enabling self-recognition and markerless reaching on a musculoskeletal robot.","key_machinery":"The load-bearing object is the dense image Jacobian $\\mathbf{J}^D$, a per-pixel array of $2 \\times N_j$ matrices where $N_j$ is the number of joints. Each pixel's Jacobian is updated by a simplified Kalman filter: the observation is the dense optical flow $u$, the observation matrix is built from the joint velocity $\\dot{q}$, and the state covariance is approximated as diagonal, so only one variance per joint is stored. The update rule $J_{k|k-1,i} \\leftarrow \\mathrm{interpolate}(J^D_{k-1},\\, x_i - J_{k-1,i}\\,\\dot{q})$ warps the previous Jacobian field by the predicted flow, which is what keeps external movement from contaminating the map. On top of this, self-recognition clusters the per-pixel Jacobian vectors with k-means and scores cluster centers by how consistent they are over time, while the servoing controller uses $\\Delta q = k_p\\, J_{\\mathrm{self}}^{\\#}(p_{\\mathrm{target}} - p_{\\mathrm{self}})$.","core_discovery":"The central claim is that the image Jacobian — the linear map from joint velocity to flow at a point — does not need to be estimated only at a few marked points. DIJE estimates it for every pixel in real time, and this single dense representation is sufficient for both recognizing the robot's own body and servoing it. The per-pixel estimator is a Kalman filter whose state is the concatenated Jacobian; by approximating the state covariance as diagonal with matched horizontal and vertical variance, memory use drops from $O(N_j^2)$ to $O(N_j)$ per pixel, making dense estimation tractable for many-joint robots. The paper's key design choice is the transport step: each pixel's Jacobian is warped to its new image location using the flow predicted from the current Jacobian estimate, not the measured optical flow, so background motion does not leak into the map. From the resulting dense field, k-means on the per-pixel Jacobian vectors plus a temporal-consistency score labels the body, and the pseudoinverse of the interpolated Jacobian at a clicked point generates joint commands.","pith_inferences":["The dense Jacobian is effectively a global visuomotor policy, so a natural extension is to use it for tasks the paper does not test, such as predicting the visual outcome of a planned trajectory or selecting which joints to recruit for a given image-space goal.","The consistency-scoring step for self-recognition is essentially slow-feature analysis applied to learned Jacobian vectors; this suggests that self-attribution may be framed generally as temporal stability of a learned sensorimotor feature, beyond vision.","Because optical flow lacks object constancy, the method breaks under occlusion, deformation, or out-of-frame motion; combining the dense Jacobian with a dense appearance-based descriptor, as the authors note, would let the map persist across such events at the cost of also estimating a deformation field.","Since the controller is purely 2D, a testable extension is to add a depth channel to the image Jacobian, which would lift the same framework to 3D servoing without changing the per-pixel estimation machinery."],"forward_implications":["A robot with no kinematic model can start from a zero-initialized dense Jacobian and learn, in real time, a complete visuomotor map of its body from its own motion.","Because the map is dense, a point that was never tracked before — such as the elbow after controlling the wrist — already has a Jacobian estimate, so the target can be switched instantly without relearning.","The same dense Jacobian field supports both perception (self/non-self segmentation) and control (servoing), and it transfers immediately to tools held in the hand.","The predicted-flow transport step prevents background motion from corrupting the self-label, even when the external motion overlaps the robot's body.","All algorithms run at 30 fps on a conventional notebook computer, so the approach is deployable online on physical robots."],"supporting_citations":[{"why":"Supplies the Kalman-filter formulation for online image Jacobian estimation that DIJE extends from a few points to every pixel.","marker":"[5]"},{"why":"Defines the classical Jacobian update rule that DIJE's simplified Kalman filter reduces to as a special case.","marker":"[1]"},{"why":"Provides the two-frame optical flow algorithm used to compute the dense flow measurements input to the estimator.","marker":"[17]"},{"why":"Describes the musculoskeletal humanoid Musashi, the physical platform on which all experiments were run.","marker":"[13]"}],"fun_headline_variants":["Dense pixel-wise Jacobian enables markerless robot control","Robot builds per-pixel self-map without markers or model","Real-time dense Jacobian estimation for robust self-recognition","No markers, no model: robot learns its body pixel by pixel"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method stands on the assumption that the linear model $u = J \\dot{q}$ holds for every pixel on the robot over a single timestep, so that warping the previous Jacobian estimate by the predicted flow moves it to the right image location; if the Jacobian is still wrong early in learning or the motion is too large, the warped estimate can lock in an incorrect but self-consistent map.","fun_headline_variants_meta":{"raw":{"variants":["Dense pixel-wise Jacobian enables markerless robot control","Robot builds per-pixel self-map without markers or model","Real-time dense Jacobian estimation for robust self-recognition","No markers, no model: robot learns its body pixel by pixel"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1319,"prompt_tokens":946,"completion_tokens":373,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":562,"tokens_out":373,"duration_ms":4632,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:14:58.259807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeatedly run the visual servoing experiment with the dense Jacobian initialized to different nonzero matrices (rather than zeros) and record the joint angles at convergence; if the robot consistently reaches the clicked target in image space but ends up in measurably different arm postures across initializations, the warped update has frozen a self-consistent but incorrect Jacobian rather than converging to the true kinematic map.","supporting_citations":[{"cited_title":"Online estimation of image jacobian matrix by Kalman-Bucy filter for uncalibrated stereo vision feedback,","cited_arxiv_id":null,"evidence_quote":"Supplies the Kalman-filter formulation for online image Jacobian estimation that DIJE extends from a few points to every pixel."},{"cited_title":"Versatile visual servoing without knowl- edge of true jacobian,","cited_arxiv_id":null,"evidence_quote":"Defines the classical Jacobian update rule that DIJE's simplified Kalman filter reduces to as a special case."},{"cited_title":"Two-Frame motion estimation based on polynomial expansion,","cited_arxiv_id":null,"evidence_quote":"Provides the two-frame optical flow algorithm used to compute the dense flow measurements input to the estimator."},{"cited_title":"Component modularized design of musculoskeletal humanoid platform musashi to investigate learning control systems,","cited_arxiv_id":null,"evidence_quote":"Describes the musculoskeletal humanoid Musashi, the physical platform on which all experiments were run."}],"review_version":1}