REVIEW 4 major objections 6 minor 28 references
Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Relative particle size and density of granular materials can be estimated from a video alone by projecting probe-dragging motion into a learned latent space.
desk verdict Genuinely new idea—video-only granular property estimation via visuo-haptic learning—but the headline claim rests on post-hoc qualitative latent inspection, not quantitative validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the probe-dragging contact model $F_d = \eta \rho g d_c H^2$, paired with the equality $\sum_i m_i \ddot{x}_i = F_d$ that ties the measured drag force to the visible accelerations of particles in the failure wedge. The network itself is an encoder-decoder: a pre-trained tracker turns each video into trajectories of sampled points, 3D convolutions compress the trajectories into a 4D latent vector, and deconvolutions reconstruct the force sequence used as supervision. After training, two of the four latent dimensions are selected as the implicit property axes for particle size and density.
What would settle it
Prepare granular samples that vary particle size and density independently (same-size beads of different materials, or sieved fractions of the same material), train the same pipeline, and check whether the two chosen latent axes correlate monotonically with independently measured size and density; any strong cross-correlation, or a large latent shift when only shape or moisture changes, would refute the claim.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a network trained to map visual particle trajectories to measured drag force spontaneously organizes its latent space by two physical properties of the grains. With a four-dimensional latent vector, two dimensions arrange the tested materials diagonally by particle size and cluster them by density; unseen materials project into compatible locations, and a handheld camera produces projections close to those from the robot arm. The authors explain the result through the probe-dragging contact model $F_d = \eta \rho g d_c H^2$, where the drag force is proportional to density $\rho$ and particle size $d_c$, so the network's force-supervised learning is expected to encode their visual correlates. The estimator is deliberately relative rather than absolute, ranking materials by projection position rather than returning calibrated physical values.
Load-bearing premise
The property-estimation claim collapses if the two latent dimensions picked after training do not independently encode particle size and density but instead encode some mixture of size, density, shape, friction, moisture, and tracking quality, since the paper's validation is visual alignment with coarse manual groupings rather than direct measurement.
Editorial extensions
If this is right
- A camera alone, including a smartphone held by hand, can rank granular materials by relative particle size and density in settings where balances, calipers, or force sensors are not available.
- Training needs no manual property labels: the measured force sequence is the supervisory signal, removing a major labor bottleneck in building granular datasets.
- The learned property distribution transfers to unseen materials whose probe-dragging behavior resembles the training regime, as shown with new lab grains and beach sands.
- Materials that violate the assumed failure-wedge behavior, such as wet sand that cracks instead of flowing, project outside the learned distribution and are effectively flagged as out of scope.
Reading between the lines
- Because the contact model couples density and size through the product $\rho d_c$, the two chosen latent axes may encode that product plus other correlated factors (shape, friction, moisture) rather than the two properties separately; an engineered set of particles varying size and density independently would settle what each axis measures.
- The same recipe—a physics-derived contact relation, force-supervised video encoding, and latent-space inspection—could transfer to other manipulations such as pouring, stirring, or scooping wherever a closed-form relation links visible motion to a material property.
- The out-of-range projections for crushed peanuts and wet sand suggest the latent space can double as an out-of-distribution detector, signaling when the underlying flow regime assumption fails.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a visuo-haptic learning framework for estimating the relative particle size and density of granular materials (GMs) from video alone. A probe is dragged through a granular medium while a camera records particle motion and a force/torque sensor records the drag force. The paper uses a simplified contact model Fd = ηρgdcH² (Eq. 1) to motivate an encoder–decoder network: the encoder processes tracked particle trajectories extracted from video, and the decoder predicts the force sequence. Training is supervised by measured force signals, avoiding manual property labels. After training, the decoder is discarded and the 4-dimensional latent space of the encoder is examined; the authors report that two of the four latent dimensions reveal an implicit property distribution in which particle size increases along one direction and density clusters in another (Fig. 6). They evaluate force prediction accuracy, compare against a classification baseline, perform ablations with and without particle tracking, test generalization to handheld-device videos and to beach sands, and select the latent dimension via validation loss (Sec. V-G). The central claim is that relative particle size and density can be estimated from a new video by projecting it into this latent space, using only a camera at inference time.
Significance. If the property-estimation claim is substantiated, the method would be a practical, low-cost tool for ranking granular materials by particle size and density without dedicated instruments, force sensors, or manual labels, with potential applications in agriculture and field geology. The paper's strengths include the use of real-world data (GM15-VF, 15 granular materials, 100 trials each), a physically motivated architecture, a genuine attempt at generalization (unseen GMs, handheld capture, beach sand), and honest reporting of failure cases. The force-inference results are reasonably convincing. However, the headline contribution—property estimation from video—is supported only by qualitative latent-space visualizations, with no quantitative metric of estimation accuracy, no error bars, and no demonstration that the selected latent dimensions robustly and independently encode size and density. This gap is load-bearing and must be addressed before the central claim can be accepted.
major comments (4)
- [Sec. V-B, Fig. 6] The property-estimation claim is validated only qualitatively. The paper reports no measured particle sizes or densities, no correlation or regression between latent projections and ground-truth properties, no classification accuracy for the size categories, and no error bars or confidence intervals for the estimated properties. For a paper whose headline is 'relative values of particle size and density can be estimated from video,' this is a load-bearing omission. Please provide quantitative evaluation on held-out GM types: e.g., Spearman rank correlation between latent axis values and measured particle size/density, R² of a linear readout from the latent space, or accuracy of size-category classification, with statistics over random seeds and train/test splits.
- [Sec. V-B and Eq. (1)] Nothing in the training objective forces the two selected latent dimensions to independently encode particle size and density. The force target Fd = ηρgdcH² depends, for fixed H and η, only on the product ρ·dc, so any latent factorization that predicts the force equally well is acceptable. The visual trajectories (Σ mi ẍi) could in principle break this symmetry because mi depends on volume as well as density, but the paper provides no check that this occurs. The selection of the two 'property' dimensions is post hoc, and Sec. V-G selects the latent dimensionality by force-prediction loss only, not by property-decoding performance. Please demonstrate that the latent axes are stable across training seeds, that other pairs of latent dimensions do not yield equally interpretable plots, and ideally that a linear decoder or simple regression can recover size and density from the latent code.
- [Sec. V-B, Sec. V-F, Sec. VI] The paper's own experiments identify two clear failure cases: crushed peanuts (Fig. 6) and beach sand GM17 with high water content (Fig. 10). The authors acknowledge these cases in Sec. VI, which is commendable, but the conclusion that the method generally estimates relative properties is thereby qualified. The paper should state the intended domain of validity (e.g., dry, non-adhesive, homogeneous granules) and report how many of the tested GM types succeed and fail under the proposed qualitative criterion, rather than presenting these as isolated exceptions. A quantitative accuracy measure would also make the boundary of applicability precise.
- [Sec. V-C] The baseline comparison does not quantitatively evaluate property estimation. The baseline is trained for GM classification, and its latent space is only visualized (Fig. 7-b); no metric compares how well the baseline or the proposed method separates properties. Since the claimed advantage is interpretability and property estimation, the comparison should include the same quantitative property-estimation metrics proposed in the first major comment, applied to both methods. As written, the baseline experiment supports a claim about latent-space scatter but not a claim about superior property estimation.
minor comments (6)
- [Sec. IV-A] The dataset section reports 100 instances per GM and an 80/10/10 split, but does not state whether the instances are independent repeated trials or sequential frames from a smaller number of drags, nor how the split was randomized. Please clarify to allow reproducibility.
- [Eq. (1)] Equation (1) equates a scalar magnitude Fd with a vector sum Σ mi ẍi; please state explicitly that only force magnitudes are considered and define the direction convention. Also, η is described as 'surface morphology' but is not defined dimensionally; a brief note that η is an empirical dimensionless coefficient would help.
- [Sec. V-B, Fig. 6] The text says particle density was 'determined by measuring the mass of the same volume.' Please specify whether this is particle density or bulk density, and whether the container tare mass was subtracted; the caption's phrase 'weights of GMs in the same container' is ambiguous.
- [Table I] The table header 'BOLD FOR LOWEST VALUES' should be reworded (e.g., 'Boldface indicates the lowest value in each column'), and the loss values should include units or a note that they are normalized MSE.
- [Sec. V-G] Table II reports only the mean validation loss for each latent dimension; given the small validation set, standard deviations or error bars should be reported to support the claim that 4 dimensions is uniquely best.
- [Sec. V-E] The handheld-device generalization experiment tests only two GMs (coffee bean and sunflower seed) and reports three projected points. This is a useful pilot demonstration, but the text should explicitly frame it as a small pilot rather than a comprehensive generalization study.
Circularity Check
No circularity in the training pipeline; the property-estimation stage is post hoc and underdetermined, but no step reduces by construction to its inputs.
full rationale
The derivation chain is not circular. The encoder-decoder is supervised by measured force sequences from an F/T sensor (Sec. III-C, Sec. IV-A), not by particle size or density labels, so the property values are not fitted inputs renamed as predictions. The contact model Fd = ηρgdcH2 is cited from external physics literature ([9], Albert et al.), not from the authors' own prior work, so the force-property relation is an independent input rather than a self-derived conclusion. Force prediction is benchmarked against held-out force measurements (Sec. V-A, Tab. I), providing an external check of the trained mapping. The property-estimation stage (Sec. V-B) selects two of four latent dimensions after training and interprets them by overlaying manually categorized sizes and container weights; this is a post hoc interpretation rather than a quantity derived from the loss. Because the force target depends on the product ρ·dc, the network is not forced by construction to factorize size and density, and the visual trajectories could in principle break that symmetry, but the paper gives no quantitative proof. That underdetermination is a correctness and validation risk, not a circular reduction. The self-citations present ([2], [11], [28]) are background and future-work references and do not carry the central argument. No equation in the paper is equivalent to its own output by definition, and no load-bearing self-citation chain is present.
Assumptions & free parameters
free parameters (1)
- latent_dim =
4
assumptions (4)
- domain assumption Contact model Fd = ηρgdcH2 from Albert et al. [9] holds for the probe-dragging setup.
- domain assumption Granular materials are homogeneous.
- domain assumption Force supervision forces the latent space to encode physically meaningful property information.
- domain assumption Particle tracking accurately follows granule motion.
Cite this review
Pith. "Pith review of Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning." pith.science (2026). https://pith.science/paper/7DJRAJGA
@misc{pith2026241202119,
author = {Pith},
title = {Pith review of: Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/7DJRAJGA}},
note = {Machine review of arXiv:2412.02119}
}
read the original abstract
Granular materials (GMs) are ubiquitous in daily life. Understanding their properties is also important, especially in agriculture and industry. However, existing works require dedicated measurement equipment and also need large human efforts to handle a large number of particles. In this paper, we introduce a method for estimating the relative values of particle size and density from the video of the interaction with GMs. It is trained on a visuo-haptic learning framework inspired by a contact model, which reveals the strong correlation between GM properties and the visual-haptic data during the probe-dragging in the GMs. After training, the network can map the visual modality well to the haptic signal and implicitly characterize the relative distribution of particle properties in its latent embeddings, as interpreted in that contact model. Therefore, we can analyze GM properties using the trained encoder, and only visual information is needed without extra sensory modalities and human efforts for labeling. The presented GM property estimator has been extensively validated via comparison and ablation experiments. The generalization capability has also been evaluated and a real-world application on the beach is also demonstrated. Experiment videos are available at \url{https://sites.google.com/view/gmwork/vhlearning} .
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Mobility of a sphere in vibrated granular media,
O. Zik, J. Stavans, and Y . Rabin, “Mobility of a sphere in vibrated granular media,” EPL (Europhysics Letters) , vol. 17, no. 4, p. 315, 1992
work page 1992
-
[2]
A Haptic-Based Proximity Sensing System for Buried Object in Granular Material
Z. Zhang, R. Jia, Y . Yan, R. Han, S. Lin, Q. Jiang, L. Zhang, and J. Pan, “A haptic-based proximity sensing system for buried object in granular material,” arXiv preprint arXiv:2411.17083 , 2024
work page Pith review arXiv 2024
-
[3]
M.-D. Yang, Y .-C. Hsu, W.-C. Tseng, C.-Y . Lu, C.-Y . Yang, M.-H. Lai, and D.-H. Wu, “Assessment of grain harvest moisture content using machine learning on smartphone images for optimal harvest timing,” Sensors, vol. 21, no. 17, p. 5875, 2021
work page 2021
-
[4]
M. W. Rasheed, J. Tang, A. Sarwar, S. Shah, N. Saddique, M. U. Khan, M. Imran Khan, S. Nawaz, R. R. Shamshiri, M. Aziz, et al. , “Soil moisture measuring techniques and factors affecting the moisture dynamics: A comprehensive review,” Sustainability, vol. 14, no. 18, p. 11538, 2022
work page 2022
-
[5]
R. Castilla-Arquillo, A. Mandow, C. J. P ´erez-del Pulgar, C. ´Alvarez- Llamas, J. M. Vadillo, and J. Laserna, “Thermal imagery for rover soil assessment using a multipurpose environmental chamber under simulated mars conditions,” IEEE Transactions on Instrumentation and Measurement, 2023
work page 2023
-
[6]
Risk assessment of geological landslide hazards using d-insar and remote sensing,
J. Zhong, Q. Li, J. Zhang, P. Luo, and W. Zhu, “Risk assessment of geological landslide hazards using d-insar and remote sensing,” Remote Sensing , vol. 16, no. 2, p. 345, 2024
work page 2024
-
[7]
Inferring the material properties of granular media for robotic tasks,
C. Matl, Y . Narang, R. Bajcsy, F. Ramos, and D. Fox, “Inferring the material properties of granular media for robotic tasks,” in 2020 ieee international conference on robotics and automation (icra) . IEEE, 2020, pp. 2770–2777
2020
-
[8]
Estimating properties of solid particles inside container using touch sensing,
X. Guo, H.-J. Huang, and W. Yuan, “Estimating properties of solid particles inside container using touch sensing,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 8985–8992
work page 2023
Show all 28 references
-
[9]
Slow drag in a granular medium,
R. Albert, M. Pfeifer, A.-L. Barab ´asi, and P. Schiffer, “Slow drag in a granular medium,” Physical review letters , vol. 82, no. 1, p. 205, 1999
1999
-
[10]
Cotracker: It is better to track together,
N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht, “Cotracker: It is better to track together,” arXiv preprint arXiv:2307.07635, 2023
2023 arXiv
-
[11]
One fling to goal: Environment-aware dynamics for goal- conditioned fabric flinging,
L. Yang, L. Yang, H. Sun, Z. Zhang, H. He, F. Wan, C. Song, and J. Pan, “One fling to goal: Environment-aware dynamics for goal- conditioned fabric flinging,” arXiv preprint arXiv:2406.14136 , 2024
2024 arXiv
-
[12]
Rigid-soft interactive learning for robust grasping,
L. Yang, F. Wan, H. Wang, X. Liu, Y . Liu, J. Pan, and C. Song, “Rigid-soft interactive learning for robust grasping,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 1720–1727, 2020
2020
-
[13]
Robot cooking with stir-fry: Bimanual non-prehensile manipulation of semi-fluid objects,
J. Liu, Y . Chen, Z. Dong, S. Wang, S. Calinon, M. Li, and F. Chen, “Robot cooking with stir-fry: Bimanual non-prehensile manipulation of semi-fluid objects,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 5159–5166, 2022
2022
-
[14]
Understanding dynamic tactile sensing for liquid property estimation,
H.-J. Huang, X. Guo, and W. Yuan, “Understanding dynamic tactile sensing for liquid property estimation,” arXiv preprint arXiv:2205.08771, 2022
2022 arXiv
-
[15]
Goats: Goal sampling adaptation for scooping with curriculum reinforcement learning,
Y . Niu, S. Jin, Z. Zhang, J. Zhu, D. Zhao, and L. Zhang, “Goats: Goal sampling adaptation for scooping with curriculum reinforcement learning,” arXiv preprint arXiv:2303.05193 , 2023
2023 arXiv
-
[16]
Deep visuo-tactile learning: Estimation of tactile properties from images,
K. Takahashi and J. Tan, “Deep visuo-tactile learning: Estimation of tactile properties from images,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 8951–8957
2019
-
[17]
Edo-net: Learning elastic properties of deformable objects from graph dynamics,
A. Longhini, M. Moletta, A. Reichlin, M. C. Welle, D. Held, Z. Erickson, and D. Kragic, “Edo-net: Learning elastic properties of deformable objects from graph dynamics,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 3875–3881
2023
-
[18]
Learning audio feedback for estimating amount and flow of granular material,
S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer, “Learning audio feedback for estimating amount and flow of granular material,” Proceedings of Machine Learning Research , vol. 87, 2018
2018
-
[19]
Uncertainty- aware self-supervised target-mass grasping of granular foods,
K. Takahashi, W. Ko, A. Ummadisingu, and S.-i. Maeda, “Uncertainty- aware self-supervised target-mass grasping of granular foods,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 2620–2626
2021
-
[20]
Generation of a tactile-based pouring motion using fingertip force sensors,
T. Kiyokawa, M. Ding, G. A. G. Ricardez, J. Takamatsu, and T. Oga- sawara, “Generation of a tactile-based pouring motion using fingertip force sensors,” in 2019 IEEE/SICE International Symposium on System Integration (SII). IEEE, 2019, pp. 669–674
2019
-
[21]
Learning robotic pow- der weighing from simulation for laboratory automation,
Y . Kadokawa, M. Hamaya, and K. Tanaka, “Learning robotic pow- der weighing from simulation for laboratory automation,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 2932–2939
2023
-
[22]
Undulatory swimming in sand: subsurface locomotion of the sandfish lizard,
R. D. Maladen, Y . Ding, C. Li, and D. I. Goldman, “Undulatory swimming in sand: subsurface locomotion of the sandfish lizard,” science, vol. 325, no. 5938, pp. 314–318, 2009
2009
-
[23]
A data-driven approach for fast simulation of robot locomotion on granular media,
Y . Zhu, L. Abdulmajeid, and K. Hauser, “A data-driven approach for fast simulation of robot locomotion on granular media,” in 2019 international conference on robotics and automation (ICRA) . IEEE, 2019, pp. 7653–7659
2019
-
[24]
Local dilation and compaction of granular materials induced by plate drag,
M. Kobayakawa, S. Miyai, T. Tsuji, and T. Tanaka, “Local dilation and compaction of granular materials induced by plate drag,” Physical Review E, vol. 98, no. 5, p. 052907, 2018
2018
-
[25]
A model for predicting soil-tool interac- tion,
W. Swick and J. Perumpral, “A model for predicting soil-tool interac- tion,” Journal of Terramechanics , vol. 25, no. 1, pp. 43–56, 1988
1988
-
[26]
Adam: A method for stochastic optimiza- tion,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimiza- tion,” arXiv preprint arXiv:1412.6980 , 2014
2014 arXiv
-
[27]
Target-free extrinsic calibration of event-lidar dyad using edge correspondences,
W. Xing, S. Lin, L. Yang, and J. Pan, “Target-free extrinsic calibration of event-lidar dyad using edge correspondences,” IEEE Robotics and Automation Letters, 2023
2023
-
[28]
Polymer-based self- calibrated optical fiber tactile sensor,
W. Chen, Y . Yan, Z. Zhang, L. Yang, and J. Pan, “Polymer-based self- calibrated optical fiber tactile sensor,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2023, pp. 10 197–10 203
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.