Pith.

Computer Vision and Pattern Recognition · arXiv:2611.16424 v1 · submitted 12 Nov 2026 · 30 references

PhysCapBench: A Cross-Domain Benchmark for Physically Grounded Video Captioning

Haofeng Wang, Pengcheng Wen, Ruiqi Li, Yuhao Zhang, Tiange Zhang, Boren Zheng, Xuanyi Liu, Siyu Tao

show 7 more authors

Danni Yu, Gan Lin, Haocheng Tang, Tianzizhe Guo, Mingyuan Su, Boyuan Chen, Zhongyuan Wang

The paper says

“Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution.”

more of the abstract

A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.

A solid benchmark for physical caption quality across three video domains; the main weakness (single LLM judge) is acknowledged and partially mitigated by downstream experiments. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 30 references say.

The paper says, at Section 5.1, Section 6.2

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.81):

“Comparisons with human ratings and additional judges would help characterize scoring consistency across domains and captioning styles. 6.3.”

Pith says: Objection 1 of 2 · major

The evaluation relies entirely on a single LLM judge (Doubao Seed 2.1 Pro) with no validation against human raters. While Section 6.2 acknowledges this limitation, it is load-bearing because the benchmark's central claim as a 'yardstick' depends on the reliability of these scores.

more of objection 1

Without inter-rater agreement, human correlation analysis, or at least a comparison with an alternative judge, the dimension scores and strategy rankings may reflect idiosyncrasies of that particular LLM rather than true physical description quality. The authors should either provide a human-rating calibration study or clearly bound the validity of their evaluation (e.g., 'scores indicate relative ordering under this judge, not absolute physical accuracy').

The paper says, at Section 5.4

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The paper states that 'accompanying improvements in PhysCapBench scores and policy success support using PhysCapBench to assess caption quality' (Table 5). However, the correlation between benchmark scores and downstream gains is reported only at the aggregate level (mean scores vs. mean success).

more of objection 2

A per-video or per-task correlation analysis (e.g., scatter plot of score vs. success change) would directly test whether the benchmark predicts downstream utility. Without this, the claim that the benchmark serves as a proxy for physical caption quality remains circumstantial.

6 minor objections

The paper says, at Section 5.2, Figure 5

Pith says: minor objection 1 of 6

The 'Physis' captioning strategy is used without explanation. The caption says 'Physis captioning is currently in internal testing.' Readers unfamiliar with the Physis-Lang paper (Ref. [14]) will be confused. Please add a brief description or a clear reference.

The paper says, at Table 4

Pith says: minor objection 2 of 6

The overall average for Physis is 79.36. Computing the mean of the seven row values gives 84.42+84.83+87.73+77.14+92.75+49.58+75.75 = 552.2 / 7 = 78.89, not 79.36. Please check the aggregation (maybe some dimensions are excluded per video). Clarify in the table footnote how the overall average is computed and why it may differ from a simple row average.

show 4 more

The paper says, at Figure 6

Pith says: minor objection 3 of 6

The y-axis label reads 'LPIPS / PVCS' but the two metrics are plotted on different scales. Adding separate small multiples or clearly labeling the left/right axes would improve readability.

The paper says, at Appendix D, Table 17

Pith says: minor objection 4 of 6

The scoring rubric bands (90–100, 75–89, etc.) are defined with phrases like 'minor omissions' and 'major omissions.' It would be helpful to provide one concrete example per band for a representative dimension (e.g., Contact and Interaction) to anchor the judge's behavior and help readers interpret the scores.

The paper says, at References

Pith says: minor objection 5 of 6

Ref. [14] (Physis-Lang) introduces a benchmark also named 'PhysCapBench.' The present paper should explicitly distinguish its benchmark from that concurrent work and discuss any differences in design or scope.

The paper says, at Section 4.1

Pith says: minor objection 6 of 6

The annotation workflow uses a 'vision–language annotator' (Figure 3). It is unclear whether this is an existing system or a custom pipeline. Please specify the model or method used for draft generation and how its outputs interact with human verification.

What would settle it

Randomly sample 100 videos balanced across third-person, egocentric, and handheld-gripper domains. Have at least three expert human raters independently score candidate captions (e.g., Meta, Physis, Cosmos) on the seven physical dimensions using the same rubric (Table 17). Compute Spearman rank correlation between human mean scores and judge scores per dimension. Report Fleiss' kappa for human agreement. If the per-dimension correlation is below 0.7 or if the judge systematically deviates on any dimension, the claim that the benchmark provides a 'yardstick' is not yet supported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees: says it holds up only with conditions 18 s · 27,961 tokens read, 2,170 written Benchmark evaluation relies on a single LLM judge not validated against human raters; unless judge scores correlate well with human assessment, the central metric lacks validity.
  3. Editor wrote the verdict above 11 s · 28,759 tokens read, 1,252 written
  4. Referee major revision: 2 major objections, 6 minor 23 s · 28,317 tokens read, 2,902 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here's my take on PhysCapBench.

The paper does what it says: it builds a benchmark for evaluating how well captions capture physical content in videos, covering three domains that actually matter for world-model and robot training—third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. That cross-domain unification is new and genuinely useful. The annotation pipeline is careful: human-verified, structured around seven physical dimensions, and documented in detail. The downstream experiments—video reconstruction and policy fine-tuning—show that physically enriched captions produce measurable gains (lower LPIPS, higher task success). Those experiments are a real strength: they independently confirm that the benchmark is measuring something that matters.

The soft spot is the one the authors themselves flag in Section 6.2: the judge is a single LLM (Doubao Seed 2.1 Pro), with no human rating comparison yet. The stress-test note is worried this is load-bearing, but I think the paper handles it proportionately. The downstream results are not judge-dependent—reconstruction fidelity and policy success are direct metrics. The judge only scores captions for the benchmark itself, and the fact that enriched instructions score higher on the judge *and* improve policy success provides some correlational support. Still, for the benchmark to be a reliable yardstick, someone needs to check judge scores against human judgments. The paper says this is future work. That's honest, but it means the benchmark is not yet fully validated as a standalone metric.

Minor point: the dataset is 500 videos per domain—fine for a starter benchmark, but coverage per domain is limited.

Bottom line: this is a well-executed, technically sound resource for the physical AI community. The judge concern is real but acknowledged, and the downstream validation buys the paper credibility. I would send it to peer review. For revision, I'd ask for a small human-rating subset or at least an analysis of judge consistency across caption styles.

For my own work, I'd not cite it directly (different subfield), but I'd bring it to reading group as an example of a solid benchmark paper.

The strongest objection

The paper's central claim is that PhysCapBench provides a 'yardstick' for physical-caption quality through seven dimension scores computed by Doubao Seed 2.1 Pro (Section 5.1). The paper acknowledges this limitation in Section 6.2, but the concern is load-bearing because the entire benchmark, all strategy comparisons, and the claimed link between benchmark scores and downstream utility (Table 5a) depend on the judge's reliability. Without calibration against human raters, we cannot rule out that the judge systematically favors certain styles (e.g., longer captions, specific phrasing) rather than reflecting true physical content. Human rating studies in prior work (e.g., VidCapBench) show that automated judges can diverge from human judgments, especially on fine-grained dimensions like contact or causality. The paper provides no inter-rater agreement metrics, no human correlation study, and no analysis of judge consistency across domains or captioning strategies. This gap means the benchmark scores may not be a reliable basis for the conclusions drawn.

The reader identified reliance on a single text judge as the weakest assumption but still recommended ACCEPT with high confidence. I agree that this is the most load-bearing concern, but I judge that it is not merely an acknowledged limitation—it is a fundamental threat to the validity of the benchmark's central metric. Without validation against human raters, every comparison between captioning strategies and every claimed correlation between benchmark scores and downstream utility rests on untested assumptions about the judge's reliability. The paper's acknowledgment in Section 6.2 does not mitigate this; it merely notes the dependency. The reconstruction experiments (Section 5.3) provide independent evidence of utility, but the benchmark itself—the main claimed contribution—is an evaluation procedure, not a dataset alone. If the judge is inconsistent or biased, the benchmark cannot serve as a dependable yardstick. Therefore, a conditional acceptance requiring a human correlation analysis is appropriate. The concrete test proposed is feasible: it requires only additional evaluation on a subset, not new data collection. The paper's other strengths (annotation pipeline, cross-domain coverage, downstream experiments) are not diminished, but the core metric needs validation.

The referee's report

If the evaluation is reliable, PhysCapBench fills a clear gap: prior benchmarks lacked multi-domain coverage and fine-grained physical dimensions. The downstream validation via reconstruction and policy tasks is a strength that goes beyond typical benchmark papers. The dataset and annotation pipeline are thorough and well-documented. The contribution is significant for the world-model and embodied AI communities.

Pith's reading of the paper (a machine's summary, not the authors')

The paper aims to establish that the physical content of video captions can be systematically measured and that such measurement predicts downstream utility in video generation and robot control. It introduces a benchmark with seven physical evaluation dimensions and human-verified annotations across 1,500 video sequences from three complementary domains. Experiments demonstrate that captions enriched with physical details reduce reconstruction perceptual distance by 9–31 percent and raise robot task success by 7.5 percentage points on a subtask of RoboTwin 2.0. The message is that language supervision for world models and embodied agents should be evaluated for its physical completeness, not just its semantic correctness.

What carries the argument. The seven physical evaluation dimensions—event coverage, objects and roles, actions and motion, contact and interaction, states and outcomes, physical mechanisms and causality, and temporal order—are the mechanism that decomposes caption quality into inspectable aspects. Each dimension is scored by a fixed text judge (Doubao Seed 2.1 Pro) against dimension-specific reference annotations that are produced through an evidence-grounded pipeline (domain-adapted observations, structured annotation, human cross-checking). The benchmark therefore provides a single protocol that can compare captioning strategies across domains and link their physical scores to downstream task outcomes.

Core claim. The central claim is that a unified benchmark spanning third-person scenes, egocentric manipulation, and handheld-gripper demonstrations can evaluate whether video captions accurately and completely describe observable physical processes, and that captions scoring higher on this benchmark also improve performance in video reconstruction and manipulation-policy fine-tuning. Specifically, physical captions reduce the learned perceptual image patch similarity (LPIPS) by 8.6–9.5% on third-person videos and 26.9–30.9% on egocentric videos relative to source metadata captions; enriching training instructions for robot policies raises PhysCapBench scores by 42 points on RoboTwin 2.0 and 39 points on RoboDojo Sim, and increases mean task success across four policy families by 0.5–2.0 percentage points per task.

The premise it rests on. The evaluation relies on a single text judge (Doubao Seed 2.1 Pro) to score captions against reference annotations; if this judge is inconsistent or biased across domains, the benchmark scores may not reflect true physical description quality.

If the paper is right (Pith's inferences, not the authors')

If the benchmark is adopted, future video-language datasets will include physical annotations as a standard quality gate, reducing the gap between video content and its language description.

World-model training pipelines that condition on captions could select or generate descriptions that maximize PhysCapBench scores, potentially improving video prediction and planning.

The seven-dimensional breakdown allows targeted caption refinement: for example, a system that scores low on contact and interaction can be prompted to add contact verbs and state-change outcomes.

Training-instruction enrichment for robot policies can be guided by per-dimension scores rather than overall success, enabling finer-grained control over what language details matter for which manipulation task.

The cross-domain design means that a single evaluation can compare physical-caption quality across third-person internet video, egocentric human demonstrations, and robotic teleoperation, supporting data mixture decisions in multi-source training.

The 7.5 percentage-point policy gain, while modest, comes from replacing only half of the training instructions with enriched text; using enriched instructions for all training steps might yield larger improvements, but the paper does not test this.

The text judge (Doubao Seed 2.1 Pro) introduces a dependency on a specific language model that is not open-source; if the judge's ranking of captions shifts with future versions, benchmark scores may not be reproducible without freezing the judge.

The benchmark's utility for guiding caption generation could be tested by iteratively optimizing a captioning model on PhysCapBench scores and then measuring transfer to an unseen downstream task—the paper does not perform this closed-loop optimization.

Because the reconstruction experiments use generated videos (not ground-truth predictions), the LPIPS reductions measure alignment between generated and reference frames, but do not directly prove that the caption causes the model to 'understand' physics—alternative explanations such as better spatial detail or lower ambiguity could also explain the improvements.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.96

“PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.94 · states limitation 0.17 · human participants 0.48 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.90 · mentions code or data availability 0.07

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. background 1.00 0.05 0.02 asserted 1.00
Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. background 1.00 0.09 0.02 asserted 1.00
A foundational data challenge is to measure and preserve this physical information as training supervision. background 0.85 0.06 0.03 asserted 1.00
In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. gap 0.80 0.13 0.05 asserted 0.97
Rich physical experience in video does not automatically translate into sufficient language supervision. gap 0.78 0.16 0.06 asserted 1.00
This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. gap 0.87 0.10 0.05 asserted 0.96
Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. contribution 1.00 0.93 0.10 asserted 0.98
It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. method 0.57 0.95 0.08 asserted 1.00
We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. method 0.60 0.91 0.05 asserted 0.87
Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. result 1.00 0.94 0.17 asserted 1.00
Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. result 1.00 0.95 0.18 asserted 1.00
PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement. contribution 0.55 0.78 0.64 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,872 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (12)
Figure 1 of the paper
Figure 1. Overview of PhysCapBench. Across three video domains, PhysCapBench provides evidence-grounded annotations and evaluates captions along seven physical dimensions. The example illustrates how a caption can describe contact while omitting shape recovery. 1 November 13, 2026 arXiv:2611.16424v1 [cs.CV] 12 Nov 2026
Figure 2 of the paper
Figure 2. Dataset overview across three domains. (a) Category composition. (b) Representative video frames and synchronized handheld-gripper views. (c) Third-person scene types, egocentric action fam￾ilies, and handheld-gripper trajectory profiles, with sequence counts. Each domain contains 500 videos or demonstration episodes
Figure 3 of the paper
Figure 3. Evidence-grounded physical annotation. (a) Domain-adapted observations guide frame se￾lection and visual inspection across third-person, egocentric, and handheld-gripper data. (b) A vision– language annotator produces physical records linked to source frames, organizing entities, events, state changes, and relations. (c) Evidence checks, human cross-review, and structural validation refine the records and captions. (d) A bottle-opening example connects hand roles, event intervals, contact transi￾tions, and object states to a physical caption. 9
Figure 4 of the paper
Figure 4. Interaction-role cues for egocentric physical annotation. Source RGB frames (left) and HOI￾DETR predictions (right). Red, yellow, and cyan boxes denote hands, directly manipulated objects, and objects affected through them; links show predicted interaction relations. Numbers beside boxes and links indicate detection and relation confidence, respectively. The role structure guides annotation of the acting hand, the manipulated object, and tool-mediated interactions, supporting the description of who acts on what. 4.4. Multi-View Handheld-Gripper Demonstrations The third domain…
Figure 5 of the paper
Figure 5. Physical caption evaluation on the third-person dataset. Physis is highlighted in black. The seven axes correspond to physical evaluation dimensions, and each curve represents a captioning strat￾egy. All strategies are evaluated against the same dimension-specific references. Scores range from 0 to 100, with higher scores farther from the center. The LingBot strategies are reconstructed from the paper description. Physis captioning is currently in internal testing
Figure 6 of the paper
Figure 6. Physical captions improve video reconstruction across domains. LPIPS and PVCS are shown for third-person (top) and egocentric (bottom) videos using H3 and LingBot-World 2.0. Original cap￾tions are source metadata captions. Point labels report mean distances; green annotations show relative reductions under physical-caption conditioning. Lower is better. Physical captions improve reconstruction in all four model–domain settings (
Figure 7 of the paper
Figure 7. Physical captions preserve interaction roles and state changes in reconstruction. Rows show source frames (top), H3 with the physical caption (middle; labeled Gold in the source experiment), and H3 with the metadata caption (bottom). Columns correspond to the same times. Physical conditioning better retains left-hand stabilization, right-hand shell removal, and the transition from a covered to an exposed interior. Metadata conditioning instead generates the assembly of a larger stool
Figure 8 of the paper
Figure 8. Physical captions preserve state transitions and interaction details in third-person recon￾struction. Columns compare the source video, H3 conditioned on the physical caption (Gold), and H3 conditioned on the metadata caption (Meta). Each row uses the same timestamp. (a) After the board is lifted, Gold retains volumetric toy shapes, while Meta exposes flattened, sheet-like shapes; the 9.50 s row shows both generated conditions after unloading. (b) Gold better retains localized drilling debris and the final separation of the drill bit from the workpiece. (c) Gold more closely…
Figure 9 of the paper
Figure 9. Robot manipulation with Training Instruction Upsampler. Frames compare a failed rollout (top) with a successful rollout from a policy fine-tuned using Training Instruction Upsampler (bottom) on RoboDojo. The failed rollout leaves bowls tilted and overturned; the fine-tuned policy completes a stable three-bowl stack. 6. Discussion and Limitations PhysCapBench connects physical caption assessment with the data and applications used in world mod￾eling. Its current scope suggests three directions for further development. 6.1. Data and Annotation Coverage The three domains provide…
Figure 10 of the paper
Figure 10. Third-person source-video examples. Rows show surface compaction, material ejection from a barrel, and compression and recovery of soft objects. Frames within each row follow temporal order, with timestamps relative to the start of the clip. Sample IDs identify the videos in the dataset manifest. (a) Unpacking and unfolding a chessboard egodex500_0073 𝑡 = 0.00 s 𝑡 = 3.70 s 𝑡 = 7.50 s 𝑡 = 11.20 s (b) Separating a shell from its base egodex500_0289 𝑡 = 0.00 s 𝑡 = 2.23 s 𝑡 = 4.50 s 𝑡 = 6.73 s (c) Disassembling a small structure egodex500_0480 𝑡 = 0.00 s 𝑡 = 2.77 s 𝑡 = 5.60 s 𝑡 = 8.37 s
Figure 11 of the paper
Figure 11. Egocentric source-video examples. Rows show unpacking and unfolding a chessboard, separating an outer shell from its base, and disassembling a small structure. The observations retain both hands and the manipulated objects across time. Timestamps and sample IDs follow the same convention as
Figure 12 of the paper
Figure 12. Handheld-gripper source-video examples. Three episodes show object placement, lid re￾moval, and textile transfer, each at two time points. Triptychs present synchronized left-wrist, high, and right-wrist views from left to right. Timestamps are measured from the start of each episode. 30

Figures are the authors', reproduced from the paper.

References (30)
  1. [1] David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018. URL https://arxiv.org/abs/1803.10122. 3 on PithPith's reviewarXiv
  2. [2] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, et al. Genie: Generative interactive environments.arXiv preprint arXiv:2402.15391, 2024. URLhttps: //arxiv.org/abs/2402.15391 on PithPith's reviewarXiv
  3. [3] NVIDIA. Cosmos world foundation model platform for physical AI.arXiv preprint arXiv:2501.03575, 2025. URLhttps://arxiv.org/abs/2501.03575. 3 on PithPith's reviewarXiv
  4. [4] Pengfei Zhou, Shengcong Chen, Di Chen, Jiaxu Wang, Rongjun Jin, Bingwen Zhu, Yike Pan, Songen Gu, Kuanning Wang, Shufeng Nan, Xingyu Qiu, Chenhao Qiu, Pu Yang, Yunuo Cai, Jianxiong Gao, Yifan Li, Yanwei Fu, Xiangyu Yue, Zhi Chen, and Jianlan Luo.𝜏 0-WM: A Unified Video- Action... on PithPith's reviewarXiv
  5. [5] Open X-Embodiment Collaboration, et al. Open X-Embodiment: Robotic learning datasets and RT- X models.arXiv preprint arXiv:2310.08864, 2023. URLhttps://arxiv.org/abs/2310.08864 on PithPith's reviewarXiv
  6. [6] Alexander Khazatsky, Karl Pertsch, Suraj Nair, et al. DROID: A large-scale in-the-wild robot ma- nipulation dataset.arXiv preprint arXiv:2403.12945, 2024. URLhttps://arxiv.org/abs/2403 .12945 on PithPith's reviewarXiv
  7. [7] Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, and Jian Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video, 2025. URLhttps://arxiv.org/abs/ 2505.11709. 4, 10 on PithPith's reviewarXiv
  8. [8] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. InRobotics: Science and Systems XX, RSS2024. Robotics: Science and Systems... on Pithdoi:10.15607/rss.2024.xx.045
  9. [9] Yang Zhou, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Haoyu Guo, Zizun Li, Kaijing Ma, Xinyue Li, Yating Wang, Haoyi Zhu, Mingyu Liu, Dingning Liu, Jiange Yang, Zhoujie Fu, Junyi Chen, Chunhua Shen, Jiangmiao Pang, Kaipeng Zhang, and Tong He. OmniWorld: A multi- domain and mult... on Pith
  10. [10] Zelin Gao, Qiuyu Wang, Jiapeng Zhu, Jingye Chen, Zichen Liu, Qingyan Bai, Jiahao Wang, Yufeng Yuan, Hanlin Wang, Yichong Lu, Ka Leong Cheng, Haojie Zhang, Jian Gao, Tianrui Feng, Yuzheng Liu, Yao Yao, Yinghao Xu, Xing Zhu, Yujun Shen, and Hao Ouyang. Infinite Worlds with Versa... on PithPith's reviewarXiv
  11. [11] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. OpenAI technical report, February 2024. URLhttps://openai.c om... on Pith
  12. [12] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70M: Captioning 70M videos with multiple cross-modality teachers. InProceedings of the IEE... on PithPith's reviewarXiv
  13. [13] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. CogVideoX: Text-to-video diffusion models with an ex... on PithPith's reviewarXiv
  14. [14] Liming Lu, Xianzheng Ma, Wenkun He, Guanqi Zhan, Yilin Zhao, Junyu Chen, Mengyao Xu, Jiao- jiao Fan, Wenhang Ge, Yuchao Gu, Yunze Liu, Boyi Li, Zhen Dong, Victor Prisacariu, Ming-Yu Liu, Song Han, and Han Cai. Physis-Lang: Self-evolving language as a physical representation fo... on PithPith's reviewarXiv
  15. [15] Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao, Yuchong Sun, Qiuyue Wang, Mingsheng Li, Sicheng Xie, Yitao Liu, Junhao Chen, et al. FineVLA: Fine-grained instruction alignment for steerable vision-language-action policies.arXiv preprint arXiv:2605.27284, 2026. 3, 5, 11 on PithPith's reviewarXiv
  16. [16] Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models, 2024. URLhttps://arxiv.org/abs/2407.00634. 3, 4 on PithPith's reviewarXiv
  17. [17] Xinlong Chen, Yuanxing Zhang, Chongling Rao, Yushuo Guan, Jiaheng Liu, Fuzheng Zhang, Chen- gru Song, Qiang Liu, Di Zhang, and Tieniu Tan. VidCapBench: A comprehensive benchmark of video captioning for controllable text-to-video generation. In Wanxiang Che, Joyce Nabende, Ekat... on Pith
  18. [18] Shi-Xue Zhang, Hongfa Wang, Duojun Huang, Xin Li, Xiaobin Zhu, and Xu-Cheng Yin. VCaps- Bench: A large-scale fine-grained benchmark for video caption quality evaluation.Proceedings of the AAAI Conference on Artificial Intelligence, 40(15):12726–12734, March 2026. ISSN 2159-539... on Pithdoi:10.1609/aaai.v40i15.38269
  19. [19] Kaixiang Qiu, Minghao Han, Keliang Liu, Yizhou Liu, Jinghang Han, Yue Jiang, Xuecheng Wu, Shunli Wang, Lihua Zhang, and Dingkang Yang. OmniFysics-Captioner technical report: Grounding omni-modal understanding in the physical world for better captioning.arXiv preprint arXiv:260... on PithPith's reviewarXiv
  20. [20] MiniMax. MiniMax H3: An open model breaking the boundaries between tasks and modalities. Official model release, July 2026. URLhttps://www.minimax.io/blog/minimax-h3. 3, 14 on Pith
show 10 more
  1. [21] Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. InInternational ... on PithPith's reviewarXiv
  2. [22] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin Le... on PithPith's reviewarXiv
  3. [23] Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. PhysBench: Bench- marking and enhancing vision-language models for physical world understanding. InInternational Conference on Learning Representations, 2025. URLhttps://physbench.github.io/. 4 on Pith
  4. [24] Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. VideoPhy: Evaluating physical common- sense for video generation. InInternational Conference on Learning Representations, 2025. URL ... on PithPith's reviewarXiv
  5. [25] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria E... on Pith
  6. [26] Macrodata Labs. WGO-Bench: What’s going on benchmark. Hugging Face dataset, 2026. URL https://huggingface.co/datasets/macrodata/WGO- Bench. Dataset card, accessed October 23, 2026. 5 on Pith
  7. [27] Ahmad Darkhalil, Dima Damen, and David Fouhey. Improving and evaluating hand-object inter- action detection.arXiv preprint arXiv:2606.17384, 2026. URLhttps://arxiv.org/abs/2606.1 on PithPith's reviewarXiv
  8. [28] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586–595, 2018. doi: 10.1109/CVPR.2018.00068. 14 on Pithdoi:10.1109/cvpr.2018.00068
  9. [29] Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles?arXiv preprint arXiv:2501.09038, 2025. URLhttps: //arxiv.org/abs/2501.09038 on PithPith's reviewarXiv
  10. [30] Wenzhen Yuan, Siyuan Dong, and Edward H. Adelson. GelSight: High-resolution robot tactile sensors for estimating geometry and force.Sensors, 17(12):2762, 2017. doi: 10.3390/s17122762. URL https://www.mdpi.com/1424-8220/17/12/2762. A. Annotation Schema and Domain Profiles The a... on Pithdoi:10.3390/s17122762

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment