REVIEW 2 major objections 5 minor 42 references
IRIS is a real 4K video benchmark that makes unsupervised recovery of physical parameters and governing equations from monocular video measurable and comparable.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-13 23:46 UTC pith:N2OTLUUS
load-bearing objection Solid real multi-body physics-from-video benchmark with SI ground truth, a usable five-axis protocol, and honest failure modes; the continuous-κ contact model is the main soft spot and is already flagged. the 2 major comments →
IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
A high-fidelity real-world benchmark of 220 videos at 4K and 60 fps spanning eight dynamical systems, three of them novel multi-body interactions, with independently measured ground-truth parameters and a standardized five-axis evaluation protocol that includes governing-equation selection, is sufficient to establish reference performance and expose systematic failure modes of latent-space and multi-step physics estimators that prior synthetic or single-body data could not reveal.
What carries the argument
The IRIS benchmark itself: controlled monocular video of single- and multi-body dynamics paired with an ODE bank, measured parameters with uncertainty, and a train-per-clip protocol that scores accuracy, equation selection, identifiability (gradient norms and residual), robustness, and extrapolation.
Load-bearing premise
Multi-body contact can be treated as a continuously active spring-like coupling inside a smooth latent ODE, rather than as true impacts with restitution and brief contact, so that existing differentiable pipelines can still be run on the new multi-body clips.
What would settle it
If a method that keeps the same latent-space pipeline but replaces the continuous coupling with a differentiable impact model recovers coupling and damping parameters with low MAE on the multi-body clips while multi-step rollout no longer diverges, the claim that the present simplified bank is an adequate test of multi-body inverse recovery would be undermined.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces IRIS, a real-world 4K/60 fps video benchmark for unsupervised physical parameter estimation and governing-equation identification from monocular video. It comprises 220 clips across eight dynamical systems (five single-body, three multi-body), with independently measured ground-truth parameters, uncertainty estimates, and a fixed train-per-clip protocol. A five-axis evaluation framework (parameter accuracy, equation selection, identifiability, robustness, extrapolation) is defined and used to evaluate a corrected latent-space baseline, a multi-step rollout loss, and four equation-routing strategies (VLM temporal reasoning, describe-then-classify, CNN classification, path-based oracle). The experiments establish reference numbers, diagnose a gradient-flow bug in a prior Euler integrator, and document systematic failure modes (multi-step instability on multi-body dynamics, damping identifiability, latent-to-SI calibration bias).
Significance. If the released artifacts match the manuscript, IRIS fills a clear gap: prior real-world data (Delfys75) is single-body only, and most inverse-physics methods are scored only on synthetic clips. The multi-body scenarios, independent GT with measurement-type labels, fixed splits, and public evaluation toolkit make the contribution reusable. Explicit diagnosis of the gradient-flow bug, the VLM ranking reversal across benchmarks, and the multi-step multi-body divergence are useful diagnostic results rather than overclaimed successes. The contact-model simplification is scoped honestly (effective κ ij, brief observability, Hitting cones excluded from MAE), so the benchmark remains a solid community resource even where the multi-body tasks are intentionally simplified.
major comments (2)
- Abstract vs. body video count: the abstract states 240 videos while Sec. 3.4, Table 2, Table S1, and the conclusion consistently report 220 videos / 22 settings. This is load-bearing for the dataset claim and must be reconciled before publication (including any held-out leaderboard partition).
- Sec. 3.4 / Appendix H multi-body contact model: the continuously active spring-like κ ij is a deliberate simplification for pipeline compatibility, but the multi-body MAE tables (Table S5) and residual diagnostics (Table 6) show that full-clip losses leave κ poorly constrained. The protocol should either (i) add contact-windowed evaluation metrics / event-conditioned losses as first-class axes, or (ii) more prominently mark multi-body parameter recovery as an open challenge rather than a primary accuracy axis, so that future users do not over-interpret absolute MAE on L and κ.
minor comments (5)
- Table 1 and Sec. 3.4: clarify that sliding-cone angle α is taken from setting metadata (hence MAE = 0.00 in Table S5) rather than recovered from video, to avoid overstating recovery performance.
- Latent-to-SI calibration (Sec. 5.3, Appendix D): state more explicitly in the main text that cross-phenomenon MAE comparisons are not strictly commensurate because of encoder-geometry-dependent heuristics.
- Fig. 1 caption and Sec. 3.4: the two multi-pendulum variants are named inconsistently in places (“two moving pendulum” / “two moving pendulum one static”); standardize labels to match Table 2.
- Sec. 6.2 footnote on g0 = 9.81: the coincidence that uncorrected baselines report MAE ≈ 0 for g is important; consider elevating a short warning into the main text so readers do not treat those zeros as successful recovery.
- Release checklist: ensure parameters.json measurement-type field ("direct" vs "fitted"), fixed split files, and the evaluation script that regenerates all tables from CSVs are present and versioned with the Hugging Face dataset.
Circularity Check
No significant circularity: IRIS is an empirical benchmark whose parameter and equation-selection metrics are scored against independently measured external ground truth, not quantities defined by the evaluated models.
full rationale
The paper’s load-bearing claims are the construction of a real-world video dataset (220 clips at 4K/60 fps, eight phenomena including multi-body interactions), independent ground-truth measurement (tape, inclinometer, laser, calipers, plus documented trajectory fits for damping/friction with uncertainty), a five-axis evaluation protocol, and diagnostic baselines that expose failure modes (gradient-flow bug, multi-step divergence on multi-body). Parameter MAE and equation-selection accuracy are computed against these external labels under a train-per-clip protocol; the multi-step loss, VLM prompts, and simplified κij contact model are experimental choices whose limitations are explicitly scoped (effective coupling only, brief observability window, Hitting cones excluded from MAE for lack of independent GT). No derivation step reduces a claimed prediction to a fitted input by construction, no uniqueness theorem is imported from overlapping authors, and self-citations (e.g., to Delfys75) are used for comparison and bug diagnosis rather than as load-bearing premises. The minor abstract/body video-count discrepancy (240 vs 220) is editorial, not circular. The work is therefore self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
free parameters (4)
- multi-step horizon K and geometric weights =
K=5, w=[1,1,0.5,0.5,0.25]
- latent dimension and encoder architecture =
d=2
- physics-parameter initialization γ0,γ1 =
(0.5, 0.05)
- train/val/test split per setting =
7/1/2
axioms (5)
- domain assumption Observed monocular video is driven by a low-dimensional latent state evolving under a known ODE family from a fixed bank.
- domain assumption Controlled laboratory conditions with fixed lighting and camera placement yield repeatable dynamics whose parameters can be independently measured.
- ad hoc to paper Multi-body contact may be replaced by a continuously active linear coupling κij for compatibility with existing latent-space pipelines.
- domain assumption Damping and friction ground truth obtained by fitting exponential envelopes or polynomial accelerations to tracked trajectories are valid reference values (with reported uncertainty).
- ad hoc to paper Per-phenomenon latent-to-SI calibration heuristics (period extraction, known object size, dimensionless ratios) convert latent coefficients into comparable physical units.
invented entities (2)
-
IRIS multi-body ODE bank with continuous coupling κij
no independent evidence
-
Five-axis IRIS evaluation protocol (accuracy, equation selection, identifiability, robustness, extrapolation)
independent evidence
read the original abstract
Unsupervised physical parameter estimation from video lacks a common benchmark: existing methods evaluate on non-overlapping synthetic data, the sole real-world dataset is restricted to single-body systems, and no established protocol addresses governing-equation identification. This work introduces IRIS, a high-fidelity benchmark comprising 240 real-world videos captured at 4K resolution and 60fps, spanning both single- and multi-body dynamics with independently measured ground-truth parameters and uncertainty estimates. Each dynamical system is recorded under controlled laboratory conditions and paired with its governing equations, enabling principled evaluation. A standardized evaluation protocol is defined encompassing parameter accuracy, identifiability, extrapolation, robustness, and governing-equation selection. Multiple baselines are evaluated, including a multi-step physics loss formulation and four complementary equation-identification strategies (VLM temporal reasoning, describe-then-classify prompting, CNN-based classification, and path-based labelling), establishing reference performance across all IRIS scenarios and exposing systematic failure modes that motivate future research. The dataset, annotations, evaluation toolkit, and all baseline implementations are publicly released.
Figures
Reference graph
Works this paper leans on
-
[1]
Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. “Discovering governing equations from data by sparse identification of nonlinear dynamical systems”. In:Proceedings of the National Academy of Sciences113.15 (2016), pp. 3932–3937
2016
-
[2]
Neural implicit representations for physical parameter inference from a single video
Florian Hofherr, Lukas Koestler, Florian Bernard, and Daniel Cremers. “Neural implicit representations for physical parameter inference from a single video”. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2023, pp. 2093–2103
2023
-
[3]
Physics-as-inverse-graphics: Unsupervised physical parameter estimation from video
Miguel Jaques, Michael Burke, and Timothy Hospedales. “Physics-as-inverse-graphics: Unsupervised physical parameter estimation from video”. In:arXiv preprint arXiv:1905.11169(2019)
Pith/arXiv arXiv 1905
-
[4]
Distilling free-form natural laws from experimental data
Michael Schmidt and Hod Lipson. “Distilling free-form natural laws from experimental data”. In:Science324.5923 (2009), pp. 81–85
2009
-
[5]
End-to-end differentiable physics for learning and control
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J. Zico Kolter. “End-to-end differentiable physics for learning and control”. In: Advances in Neural Information Processing Systems (NeurIPS)31 (2018)
2018
-
[6]
Reasoning-modulated representations
Petar Veliˇ ckovi´ c, Matko Boˇ snjak, Thomas Kipf, et al. “Reasoning-modulated representations”. In:Learning on Graphs Conference. PMLR. 2022, pp. 50–1
2022
-
[7]
Learning physics from video: Unsupervised physical parameter estimation for continuous dynamical systems
Alejandro Casta˜ neda Garcia, Jan Warchocki, Jan van Gemert, Daan Brinks, and Nergis Tomen. “Learning physics from video: Unsupervised physical parameter estimation for continuous dynamical systems”. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025, pp. 27924–27933
2025
-
[8]
Vid2Param: Modeling of dynamics parameters from video
Martin Asenov, Michael Burke, Daniel Angelov, Todor Davchev, Kartic Subr, and Subramanian Ramamoorthy. “Vid2Param: Modeling of dynamics parameters from video”. In:IEEE Robotics and Automation Letters5.2 (2019), pp. 414–421
2019
-
[9]
Visual interaction networks: Learning a physics simulator from video
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. “Visual interaction networks: Learning a physics simulator from video”. In:Advances in Neural Information Processing Systems (NeurIPS)30 (2017)
2017
-
[10]
Galileo: Perceiving physical object properties by integrating a physics engine with deep learning
Jiajun Wu, Ilker Yildirim, Joseph J. Lim, Bill Freeman, and Josh Tenenbaum. “Galileo: Perceiving physical object properties by integrating a physics engine with deep learning”. In:Advances in Neural Information Processing Systems (NeurIPS)28 (2015). IRIS 17
2015
-
[11]
Learning to see physics via visual de-animation
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. “Learning to see physics via visual de-animation”. In:Advances in Neural Information Processing Systems (NeurIPS)30 (2017)
2017
-
[12]
Learning physics constrained dynamics using autoencoders
Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. “Learning physics constrained dynamics using autoencoders”. In:Advances in Neural Information Processing Systems (NeurIPS)35 (2022), pp. 17157–17172
2022
-
[13]
Unsupervised learning of latent physical properties using perception-prediction networks
David Zheng, Vinson Luo, Jiajun Wu, and Joshua B. Tenenbaum. “Unsupervised learning of latent physical properties using perception-prediction networks”. In:arXiv preprint arXiv:1807.09244(2018)
Pith/arXiv arXiv 2018
-
[14]
Hamiltonian generative networks
Peter Toth, Danilo Jimenez Rezende, Andrew Jaegle, S´ ebastien Racani` ere, Aleksandar Botev, and Irina Higgins. “Hamiltonian generative networks”. In: arXiv preprint arXiv:1909.13789(2019)
Pith/arXiv arXiv 1909
-
[15]
Neural ordinary differential equations
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. “Neural ordinary differential equations”. In:Advances in Neural Information Processing Systems (NeurIPS). Vol. 31. 2018, pp. 6571–6583
2018
-
[16]
Hamiltonian neural networks
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. “Hamiltonian neural networks”. In:Advances in Neural Information Processing Systems (NeurIPS). Vol. 32. 2019, pp. 15353–15363
2019
-
[17]
Lagrangian neural networks
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. “Lagrangian neural networks”. In:ICLR Workshop on Integration of Deep Neural Models and Differential Equations. 2020
2020
-
[18]
Learning to simulate complex physics with graph networks
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. “Learning to simulate complex physics with graph networks”. In:Proceedings of the International Conference on Machine Learning (ICML). Vol. 119. PMLR. 2020, pp. 8459–8468
2020
-
[19]
Interaction networks for learning about objects, relations and physics
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. “Interaction networks for learning about objects, relations and physics”. In:Advances in Neural Information Processing Systems (NeurIPS). Vol. 29. 2016, pp. 4502–4510
2016
-
[20]
Physics 101: Learning physical object properties from unlabeled videos
Jiajun Wu, Joseph J. Lim, Hongyi Zhang, Joshua B. Tenenbaum, and William T. Freeman. “Physics 101: Learning physical object properties from unlabeled videos”. In:British Machine Vision Conference (BMVC). 2016
2016
-
[21]
VideoPhy: Evaluating physical commonsense for video generation
Hritik Bansal, Zongyu Lin, Tianyi Xie, et al. “VideoPhy: Evaluating physical commonsense for video generation”. In:arXiv preprint arXiv:2406.03520(2024)
Pith/arXiv arXiv 2024
-
[22]
Do generative video models understand physical principles?
Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. “Do generative video models understand physical principles?” In:Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2026, pp. 948–958
2026
-
[23]
Chenyu Zhang, Daniil Cherniavskii, Antonios Tragoudaras, et al.Morpheus: Benchmarking Physical Reasoning of Video Generative Models with Real Physical Experiments. 2025. arXiv:2504.02918 [cs.CV].url: https://arxiv.org/abs/2504.02918
Pith/arXiv arXiv 2025
-
[24]
Meng-Hao Guo, Jiajun Xu, Yi Zhang, et al.R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation. 2025. arXiv:2505.02018 [cs.CV].url: https://arxiv.org/abs/2505.02018
Pith/arXiv arXiv 2025
-
[25]
Ao Liang, Lingdong Kong, Tianyi Yan, et al.WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World. 2025. arXiv:2512.10958 [cs.CV].url:https://arxiv.org/abs/2512.10958. 18 Khanbayov, Barhdadi et al
Pith/arXiv arXiv 2025
-
[26]
ChatGPT and open-AI models: A preliminary review
Konstantinos I. Roumeliotis and Nikolaos D. Tselikas. “ChatGPT and open-AI models: A preliminary review”. In:Future Internet15.6 (2023), p. 192
2023
-
[27]
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. “Visual instruction tuning”. In:Advances in Neural Information Processing Systems (NeurIPS)36 (2023), pp. 34892–34916
2023
-
[28]
LLaVA-Video: Video instruction tuning with synthetic data
Yuanhan Zhang, Jinming Wu, Wei Li, et al. “LLaVA-Video: Video instruction tuning with synthetic data”. In:arXiv preprint arXiv:2410.02713(2024)
Pith/arXiv arXiv 2024
-
[29]
Physics context builders: A modular framework for physical reasoning in vision-language models
Vahid Balazadeh, Mohammadmehdi Ataei, Hyunmin Cheong, Amir Hosein Khasahmadi, and Rahul G. Krishnan. “Physics context builders: A modular framework for physical reasoning in vision-language models”. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2025, pp. 7318–7328
2025
-
[30]
Mimicking the Physicist’s Eye: A VLM-centric Approach for Physics Formula Discovery
Jiaqi Liu, Songning Lai, Pengze Li, et al. “Mimicking the Physicist’s Eye: A VLM-centric Approach for Physics Formula Discovery”. In:arXiv preprint arXiv:2508.17380(2025)
Pith/arXiv arXiv 2025
-
[31]
Data-driven discovery of coordinates and governing equations
Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. “Data-driven discovery of coordinates and governing equations”. In:Proceedings of the National Academy of Sciences116.45 (2019), pp. 22445–22451
2019
-
[32]
Ruikun Li, Yan Lu, Shixiang Tang, Biqing Qi, and Wanli Ouyang.MLLM-based Discovery of Intrinsic Coordinates and Governing Equations from High-Dimensional Data. 2025. arXiv:2505.11940 [cs.CE]
Pith/arXiv arXiv 2025
-
[33]
ContactGaussian-WM: Learning Physics-Grounded World Model from Videos
Meizhong Wang, Wanxin Jin, Kun Cao, Lihua Xie, and Yiguang Hong. ContactGaussian-WM: Learning Physics-Grounded World Model from Videos
-
[34]
arXiv:2602.11021 [cs.RO].url:https://arxiv.org/abs/2602.11021
-
[35]
Yiren Song, Xiaokang Liu, and Mike Zheng Shou.DiffSim: Taming Diffusion Models for Evaluating Visual Similarity. 2024. arXiv: 2412.14580 [cs.CV].url: https://arxiv.org/abs/2412.14580
Pith/arXiv arXiv 2024
-
[36]
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis wyffels.A Differentiable Physics Engine for Deep Learning in Robotics. 2018. arXiv: 1611.01652 [cs.NE].url:https://arxiv.org/abs/1611.01652
Pith/arXiv arXiv 2018
-
[37]
Karen Liu.Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact
Keenon Werling, Dalton Omens, Jeongseok Lee, Ioannis Exarchos, and C. Karen Liu.Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact. 2021. arXiv:2103.16021 [cs.RO].url: https://arxiv.org/abs/2103.16021
Pith/arXiv arXiv 2021
-
[38]
How to train your neural ODE: the world of Jacobian and kinetic regularization
Chris Finlay, J¨ orn-Henrik Jacobsen, Levon Nurbekyan, and Adam M Oberman. How to train your neural ODE: the world of Jacobian and kinetic regularization
-
[39]
arXiv:2002.02798 [stat.ML].url: https://arxiv.org/abs/2002.02798
Pith/arXiv arXiv 2002
-
[40]
Sifan Wang, Hanwen Wang, and Paris Perdikaris.Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets
-
[41]
arXiv:2103.10974 [cs.LG].url:https://arxiv.org/abs/2103.10974
-
[42]
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, et al.SAM 2: Segment Anything in Images and Videos. 2024. arXiv:2408.00714 [cs.CV]. IRIS 19 Supplementary Material IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video A Dataset Statistics and Release Details Table S1 provides a complete per-pheno...
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.