Robotics · arXiv:2611.16423 v2 · submitted 12 Nov 2026
VOKE: Visual Object Kinematics Estimator for Dexterous In-Hand Manipulation
The paper says
“Vision-based dexterous in-hand manipulation requires knowing not only where an object is, but also how it moves.”
more of the abstract
However, most robotic manipulation systems rely on off-the-shelf visual perception models whose outputs are limited to static object masks, geometry, or 6D poses, leaving downstream controllers to infer object dynamics implicitly from noisy visual observations. We present VOKE, a kinematics-aware perception framework that augments existing object and hand-object pose estimators to recover a complete object kinematic state comprising pose, linear and angular velocity, and linear and angular acceleration. VOKE contains two core perception modules and a downstream application study. First, a Pluggable Kinematics Module (PKM) attaches to the output of a standard pose estimator and predicts the object's linear and angular velocity and acceleration from temporal visual features. Second, a Kinematics-guided Pose Refinement Module (KPRM) converts this motion vector into a second-order, motion-prior reference pose and refines the per-frame pose estimates. We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing. As a downstream application, we use Kinematics-Augmented Reinforcement Learning (KARL) to compare a kinematics-augmented policy with a pose-history baseline. After training the policies in simulation, we transfer the kinematics-augmented policy to the physical robot without policy fine-tuning, demonstrating the complete perception-to-control pathway in the target domain.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. VOKE is a well-engineered, modular framework for estimating object velocity and acceleration from video, with convincing simulation results and real-world transfer; the main caveat is that its real-dataset evaluations rely on spline-derived kinematics references whose accuracy is not independently validated. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section IV-A, Paragraph 4 (Motion-quantity reference values); Table II
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The ground truth velocity and acceleration for the DexYCB and OakInk datasets are derived by fitting cubic splines to the ground truth pose trajectories and differentiating. The authors acknowledge this (Section IV-A) and validate by replaying in MuJoCo to check 'physical meaningfulness'.
more of objection 1
However, this does not verify that the spline derivatives match the true physical velocities and accelerations. An overly smoothed spline could produce incorrect accelerations while remaining physically plausible. The simulation dataset (Isaac Sim) provides ground truth, but uses a cube with different dynamics and occlusion patterns. The paper should quantify the spline fit error (e.g., residual between fitted spline and original marker-based pose) and discuss how sensitive the results are to the choice of spline smoothing parameter. Without this, the claimed improvements in velocity/acceleration estimation over pose differencing on real datasets are not convincingly supported.
The paper says, at Table III; Table II (simulation dataset)
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The simulation dataset results (Table II, bottom row) show VOKE outperforming pose differencing. However, the reference values for the simulation are derived from the simulator (ground truth velocity) plus spline fitting for acceleration. The paper should state whether the reported 'simulation' results on velocity/acceleration use simulator-provided ground truth for velocity (and thus acceleration derived by spline).
more of objection 2
If acceleration is derived by spline, the same issue applies, albeit the underlying velocity is known exactly. Please clarify and consider using analytical derivatives for acceleration from the simulator if available.
The paper says, at Section III-A; Table V
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The PKM uses the last k frames of image features. The abaltion shows that k=3 is best. However, the KPRM also uses the same k frames of pose history, and the reference pose is computed from the previous frame's network pose. This couples the frame rate to the motion prior. The paper should discuss the sensitivity to the frame rate (Δt) in Eq. (1), especially for the second-order term, and whether this was tuned or fixed.
4 minor objections
The paper says, at Section I, line 70-73
Pith says: minor objection 1 of 4
The phrase 'substantially reduces velocity and acceleration estimation error compared with pose differencing' appears in the abstract/introduction before the method is described. Consider rephrasing to indicate that this claim is validated later.
The paper says, at Section IV-A, last paragraph
Pith says: minor objection 2 of 4
The description of the MuJoCo replay validation is vague: 'the replayed trajectories aligned well with the annotated trajectories.' Please provide a quantitative measure (e.g., positional error over time) to substantiate the claim.
show 2 more
The paper says, at Table I, footnote
Pith says: minor objection 3 of 4
The footnote says 'Values absent from the original publications are recomputed from our inference outputs where possible.' It would be helpful to specify which metrics are recomputed versus taken from published tables, espeically for the 'orig.' rows.
The paper says, at Section IV-C, sentence
Pith says: minor objection 4 of 4
In the sentence 'Every VOKE NRMSE remains below 1, whereas Pose Diff (SG) exceeds 1 for every angular-velocity and acceleration quantity and for all four quantities in simulation,' note that NRMSE exceeding 1 indicates the estimate is worse than predicting the mean; this is a useful interpretation but should be stated explicitly.
What would settle it
For a random subset of frames in DexYCB and OakInk, compute the RMS residual of the cubic spline fit to the ground truth translation and rotation trajectories. For translation: compute the L2 norm between the fitted spline at each frame and the original marker-based pose. For angular velocity: compute the residual between the splined angular velocity and a centered difference of the original poses (without splining). Report these residuals relative to the magnitude of the reference velocity/acceleration. If residuals are an order of magnitude smaller than the reported MEE, the spline-derived references are reliable; otherwise, the claimed error reduction may be partly due to smoothing artifacts.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The paper tackles a real practical gap: most pose estimators only output static pose, leaving downstream controllers to differentiate noisy estimates for velocity and acceleration. VOKE adds a lightweight, pluggable module that predicts the full kinematic state directly from temporal visual features, and uses that state for pose refinement and as policy input in RL. The engineering is solid — three backbones, two real datasets, one simulation dataset, extensive ablations, and a real-robot qualitative demo. The improvements over pose differencing are large and consistent across all metrics reported.
The soft spot is the ground truth for velocity/acceleration on the real datasets (DexYCB, OakInk). Those values come from cubic splines fitted to ground-truth pose trajectories and then differentiated. The authors try to validate this with a MuJoCo replay, but that only checks physical plausibility, not derivative accuracy. If the spline smooths out rapid motions (common under occlusion or contact), the learned PKM might be fitting to smoothed values rather than true kinematics. The simulation dataset (Isaac Sim) provides exact ground truth and shows VOKE also outperforms pose differencing there, which partially mitigates the concern — but that simulation uses a cube object with different dynamics than the real hand-object interactions. So the central claim of substantial error reduction on real benchmarks rests on an assumption that is not fully quantified.
The authors acknowledge the limitation for acceleration under rapid contacts, and they release code and data, which is good. I think the paper deserves a serious referee. The method is useful, the experiments are extensive, and the modular design is practical. The main thing a reviewer should push on is a quantitative analysis of the spline fit error (e.g., residuals, comparison with optical flow on known test motions) to bound how much of the reported gain might be due to the reference rather than true physical accuracy. That said, for the simulation and pose-refinement parts, the results stand on their own.
Verdict: worth engaging with. It's a competent, well-scoped contribution that advances the state of open tools for dexterous manipulation. I'd bring it to a reading group focused on perception-for-robotics.
The strongest objection
The paper's central claim of substantial reduction in velocity and acceleration estimation error on DexYCB and OakInk relies on spline-derived reference values for training and evaluation (Section IV-A). The only validation of these references is a MuJoCo replay that checks physical plausibility — e.g., that the object motion could be produced by physically consistent forces — but this does not verify derivative accuracy. A spline that overly smooths rapid motions could produce incorrect accelerations while remaining physically plausible, and the replay test would not detect this. The simulation dataset (Isaac Sim) provides accurate ground truth from the simulator and shows VOKE outperforms differencing, but that evaluation uses a cube object with different dynamics and occlusion patterns than the real datasets. Without quantifying the spline fit error (e.g., residual between fitted spline and original marker-based pose), the reader cannot assess whether the reported error reductions reflect true physical accuracy or merely agreement with the chosen spline model. This is load-bearing because the velocity/acceleration results are a primary contribution and the main evidence for the claimed superiority over pose differencing on real benchmarks.
The reader's weakest_assumption identifies the same concern: spline-derived ground truth may introduce error. However, the simulation results provide partial support by showing VOKE works with accurate derivatives. Still, the real dataset results are the primary evidence for the central claim, and they lack independent validation of the derivative references. The concrete test proposed would strengthen the evidence without requiring new data collection. The verdict remains CONDITIONAL (unchanged) because the reader's condition already calls for this validation; no change needed yet.
The referee's report
If the results hold, VOKE provides a practical way to endow existing pose estimators with direct kinematical state estimation, which could improve control and planning in robotic manipulation. The modular design (PKM, KPRM) and the systematic evaluation across multiple backbones and datasets are strengths. The work also demonstrates a complete perception-to-control pipeline in a dexterous manipulation task, with sim-to-real transfer, which is valuable. However, the significance is moderated by reliance on spline-derived ground truth for velocity/acceleration on real datasets, which undermines the strength of the comparative results on those benchmarks.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that predicting object velocity and acceleration directly from temporal visual features, rather than differentiating noisy pose estimates, yields a more accurate and temporally consistent kinematic state. This state can be used both to refine per-frame pose estimates and to provide a better observation for reinforcement learning policies. The key claim is that the plaggable kinematics module (PKM) and kinematics-guided pose refinement module (KPRM) together let standard pose estimators produce a complete state (pose, velocity, acceleration) that is consistently more accurate than pose differencing. The authors demonstrate this on two hand-object interaction benchmarks and show that a reinforcement learning policy using this state achieves higher success in dexterous in-hand manipulation.
What carries the argument. The framework has two core modules: a Pluggable Kinematics Module (PKM) that shares the image encoder of a base pose estimator, concatenates temporal features from the last few frames, and decodes the motion vector (linear and angular velocity and acceleration) via a lightweight MLP; and a Kinematics-guided Pose Refinement Module (KPRM) that uses this motion vector together with the previous network pose to construct a second-order motion-prior reference pose (Equation 1) and then refines the per-frame pose using a small MLP that also receives the historical pose estimates. The refined pose and motion vector constitute the complete kinematic state consumed by the downstream Kinematics-Augmented Reinforcement Learning (KARL) policy.
Core claim. The central discovery is that a lightweight module sharing the image encoder of a standard pose estimator and processing a short temporal window of features can predict linear and angular velocity and acceleration with lower error than third-order Savitzky-Golay differentiation of the estimated pose sequence. Additionally, the predicted motion vector can be integrated via a second-order motion prior to refine the per-frame pose, improving both spatial accuracy and temporal consistency. The complete kinematic state formed by refined pose and motion vector then provides a more informed observation for a reinforcement learning policy, leading to higher success rates in dexterous in-hand cube reorientation.
The premise it rests on. The ground truth velocity and acceleration for the real-world datasets are derived by fitting cubic splines to the ground truth pose trajectories and differentiating analytically; the training and evaluation rely on the assumption that these spline-based derivatives are sufficiently accurate to serve as reference values, whereas the true physical kinematics are unknown.
If the paper is right (Pith's inferences, not the authors')
Standard pose estimators such as gSDF, HFL-Net, and HOISDF can be augmented with PKM to output a complete kinematic state without architectural changes.
Downstream robotic controllers can use explicit velocity and acceleration observations, potentially reducing the need for temporal-history stacking or approximate differentiation.
The kinematic state improves temporal consistency of pose estimates: translational relative pose error is reduced by 41–64% and rotational relative pose error by 7.9–33.3% across backbones and datasets.
A reinforcement learning policy trained with the complete kinematic state in simulation converges faster and achieves higher success (78.90% vs 68.20%) than a policy using only pose history.
The simulation-trained policy can be deployed zero-shot on a physical robot by replacing simulator state with online VOKE predictions, demonstrating a closed perception-to-control pathway.
The method's reliance on spline-derived ground-truth velocity and acceleration means that on real-world data with noisy or missing ground-truth poses, the quality of the reference kinematics may degrade; true measured motion from high-speed tracking or inertial sensors could provide stronger validation.
Because PKM shares the backbone encoder, advances in pose estimation (e.g., better occlusion handling or temporal tracking) could directly improve kinematics estimation, making the approach a natural complement to ongoing work in 6D pose tracking.
The framework might generalize beyond hand-object interaction to general rigid-body manipulation (e.g., pick-and-place, stacking) where velocity and acceleration are beneficial for control or dynamics prediction.
The ablation study shows that the explicit motion vector and the second-order reference pose each contribute independently to the pose improvement; further gains might come from replacing the simple MLP refiner with a learned dynamics filter such as a Kalman filter or neural ODE.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Vision-based dexterous in-hand manipulation requires knowing not only where an object is, but also how it moves. However, most robotic manipulation systems rely on off-the-shelf visual perception models whose outputs are limited to static object masks, geometry, or 6D poses, leaving downstream controllers to infer object dynamics implicitly from noisy visual observations. We present VOKE, a kinematics-aware perception framework that augments existing object and hand-object pose estimators to recover a complete object kinematic state comprising pose, linear and angular velocity, and linear and angular acceleration. VOKE contains two core perception modules and a downstream application study. First, a Pluggable Kinematics Module (PKM) attaches to the output of a standard pose estimator and predicts the object's linear and angular velocity and acceleration from temporal visual features. Second, a Kinematics-guided Pose Refinement Module (KPRM) converts this motion vector into a second-order, motion-prior reference pose and refines the per-frame pose estimates. We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing. As a downstream application, we use Kinematics-Augmented Reinforcement Learning (KARL) to compare a kinematics-augmented policy with a pose-history baseline. After training the policies in simulation, we transfer the kinematics-augmented policy to the physical robot without policy fine-tuning, demonstrating the complete perception-to-control pathway in the target domain.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 0.84
“We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.12 · human participants 0.02 · controlled comparison 0.97 · randomized assignment 0.02 · states quantitative result 0.10 · compares with prior results 0.97 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Vision-based dexterous in-hand manipulation requires knowing not only where an object is, but also how it moves. | background 1.00 | 0.05 | 0.04 | asserted 1.00 |
| However, most robotic manipulation systems rely on off-the-shelf visual perception models whose outputs are limited to static object masks, geometry, or 6D poses, leaving downstream controllers to infer object dynamics implicitly from noisy visual observations. | gap 0.90 | 0.06 | 0.03 | asserted 1.00 |
| We present VOKE, a kinematics-aware perception framework that augments existing object and hand-object pose estimators to recover a complete object kinematic state comprising pose, linear and angular velocity, and linear and angular acceleration. | contribution 0.99 | 0.93 | 0.28 | asserted 1.00 |
| VOKE contains two core perception modules and a downstream application study. | contribution 0.80 | 0.77 | 0.04 | asserted 0.99 |
| First, a Pluggable Kinematics Module (PKM) attaches to the output of a standard pose estimator and predicts the object's linear and angular velocity and acceleration from temporal visual features. | method 0.97 | 0.86 | 0.05 | asserted 1.00 |
| Second, a Kinematics-guided Pose Refinement Module (KPRM) converts this motion vector into a second-order, motion-prior reference pose and refines the per-frame pose estimates. | method 0.98 | 0.89 | 0.05 | asserted 1.00 |
| We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing. | result 1.00 | 0.97 | 0.38 | asserted 1.00 |
| As a downstream application, we use Kinematics-Augmented Reinforcement Learning (KARL) to compare a kinematics-augmented policy with a pose-history baseline. | contribution 0.81 | 0.95 | 0.06 | asserted 0.93 |
| After training the policies in simulation, we transfer the kinematics-augmented policy to the physical robot without policy fine-tuning, demonstrating the complete perception-to-control pathway in the target domain. | result 0.85 | 0.96 | 0.25 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 7,090 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (4)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment