pith. sign in

arxiv: 2605.03941 · v2 · submitted 2026-05-05 · 💻 cs.CV · cs.AI

iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Framework

Jianjie Fang , Yingshan Lei , Qin Wan , Ziyou Wang , Yuchao Huang , Yongyan Xu , Baining Zhao , Weichen Zhang
show 3 more authors
Chen Gao Xinlei Chen Yong Li
This is my paper

Pith reviewed 2026-05-08 18:44 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords world modelsbenchmarkinteractive agentsaction generation frameworkvideo datasetphysical interactionmemory taskstrajectory following
0
0 comments X

The pith

iWorld-Bench supplies a 330k-clip dataset and unified action framework to test world models on physical interaction tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds iWorld-Bench to give world models standardized ways to train and be measured on abilities such as distance perception and memory that current work lacks at scale. It starts with 330k video clips, selects 2.1k high-quality examples across varied conditions, and introduces an Action Generation Framework that produces consistent actions no matter how each model encodes them. Six task types are defined to produce 4.9k test samples that jointly check visual generation, trajectory following, and memory. The authors then evaluate 14 existing models, surface their specific weaknesses, and release a public leaderboard.

Core claim

iWorld-Bench consists of a diverse video dataset drawn from 330k clips and reduced to 2.1k curated samples together with an Action Generation Framework that unifies evaluation across models with different interaction modalities. The framework supports six task types that assess visual generation, trajectory following, and memory through 4.9k test samples. Evaluation of 14 representative world models on this benchmark identifies key limitations in current approaches and supplies a public leaderboard for continued comparison.

What carries the argument

The Action Generation Framework that translates outputs from models with differing action modalities into a common format for consistent testing across tasks.

If this is right

  • Models can be compared fairly on interaction abilities regardless of their internal action representations.
  • The 330k clips and 2.1k samples supply training data that targets specific gaps in perception and memory.
  • The six tasks isolate weaknesses in visual generation, path following, and recall for targeted model improvement.
  • The public leaderboard enables ongoing tracking of progress as new world models appear.
  • Insights from the 14-model evaluation point to concrete directions for addressing current shortcomings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Strong results on the benchmark may indicate better transfer to downstream control problems such as robotic planning, though this remains untested.
  • The curation method from large raw video to focused high-quality samples could be reused to build narrower benchmarks for manipulation or navigation.
  • Adding tasks with longer temporal horizons might expose further memory limitations not captured by the current six types.
  • Pairing the benchmark with real-world robot trajectories would allow direct measurement of sim-to-real gaps.

Load-bearing premise

The 2.1k curated samples and six task types sufficiently represent the core physical interaction capabilities needed to advance toward AGI-level agents.

What would settle it

A model that achieves high scores across all six tasks but cannot perform the same distance, memory, or trajectory tasks when transferred to a physical robot or different simulation environment would falsify the benchmark's usefulness.

Figures

Figures reproduced from arXiv: 2605.03941 by Baining Zhao, Chen Gao, Jianjie Fang, Qin Wan, Weichen Zhang, Xinlei Chen, Yingshan Lei, Yong Li, Yongyan Xu, Yuchao Huang, Ziyou Wang.

Figure 1
Figure 1. Figure 1: Overview of iWorld-Bench. iWorld-Bench encompasses four distinct perspectives: Unmanned Ground Vehicles (UGVs), Unmanned Aerial Vehicles (UAVs), humans, and robotics. It incorporates nine types of outdoor weather conditions, five different indoor lighting conditions, thousand of diverse scenes, and thousands of entities, providing a comprehensive and diverse evaluation environment. The benchmark leverages … view at source ↗
Figure 2
Figure 2. Figure 2: Data Processing Pipeline and Overview. As shown in view at source ↗
Figure 3
Figure 3. Figure 3: Radar charts of model performance across evaluation metrics. (a) Performance comparison of all 14 models on Action Control and Memory Ability tasks across 8 metrics. (b) Performance of 7 camera-parameter-controlled models on Camera Following tasks. 5. Conclusion and Future Work iWorld-Bench is a unified benchmark specifically de￾signed for interactive world models, integrating a multi￾dimensional evaluatio… view at source ↗
Figure 4
Figure 4. Figure 4: The systematic data curation pipeline: transforming raw multi-source inputs into high-fidelity training data via structured standardization and trajectory rectification. 17 view at source ↗
Figure 5
Figure 5. Figure 5: A detailed showcase of the Difference Verification task across four difficulty levels (rows 1 to 4), based on camera operation complexity: Row 1 shows basic single-axis movements; Row 2 adds combined translation and rotation; Row 3 features sequential composite trajectories; and Row 4 involves complex multi-axis movements with view changes. These examples demonstrate the model’s ability to detect subtle po… view at source ↗
Figure 6
Figure 6. Figure 6: Detailed showcase of the Memory Verification task focusing on loop closure difficulty. This figure illustrates memory-dependent trajectories where the camera performs reversible actions (e.g., ”up then down” or ”turn right then left”), requiring the model to recall the initial state to verify the loop closure. The red bounding boxes highlight key visual cues used for temporal reasoning and consistency chec… view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of Subject Consistency of Image Quality. Image quality measures the memory symmetry. when the model execute memory tasks. (a) has a clear distortion. (b) shows a better color image quality consistency. 2. Brightness Consistency (SBrightness). This metric aims to evaluate the stability of the brightness distribution in generated videos. We categorize pixel grayscales into three levels (dark, m… view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of Subject Consistency of Brightness. We defined brightness consistency to measure the performance of light comprehensive of models. (a) has a obvious change while (b) hold great consistency throughout. The three-level classification mechanism reserves space for reasonable brightness changes, while the comparison mechanism with the initial frame effectively monitors style collapse and brightn… view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of Subject Consistency of Color Temperature. We use color temperature consistency to measure the color of pictures. (a) changes from a cold tone to a warm tone. (b) has a better color temperature consistency. 4. Sharpness Retention (SSharpness). This metric evaluates the stability of details by monitoring the evolution of edge gradients. We propose a vectorized Tenengrad method: independently… view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of Subject Consistency of Sharpness Retention. We defined sharpness retention consistency to demonstrate the clarity of the pictures. (a) is blurry through time. (b) is clearer as the bricks on the wall can be clearly seen. 5. Motion Smoothness (SMotion). This metric evaluates the sequence coherence of generated videos using the motion prior of a video interpolation model. We adopt a samplin… view at source ↗
Figure 11
Figure 11. Figure 11: Visualization of Subject Consistency of Trajectory Accuracy. Trajectory Accuracy is a indicator that shows the ability of the model to accurately follow given route. (a) could not follow accurately. (b) shows better better accuracy. (a) Inconsistent Trajectory Tolerance (score 35.21%) (b) Consistent Trajectory Tolerance (score 70.01%) view at source ↗
Figure 12
Figure 12. Figure 12: Visualization of Subject Consistency of Trajectory Tolerance. Trajectory tolerance measures model’s ability to tolerant the route not accurately designed. (a) can not follow the route smoothly while (b) has a trajectory tolerance with a better performance. 29 view at source ↗
Figure 13
Figure 13. Figure 13: Visualization of Subject Consistency of Trajectory Alignment. Trajectory alignment measures the capabilities of the model following the memory route. (a) has a fold line while (b) has a great consistency throughout memory task. 30 view at source ↗
read the original abstract

Achieving Artificial General Intelligence (AGI) requires agents that learn and interact adaptively, with interactive world models providing scalable environments for perception, reasoning, and action. Yet current research still lacks large-scale datasets and unified benchmarks to evaluate their physical interaction capabilities. To address this, we propose iWorld-Bench, a comprehensive benchmark for training and testing world models on interaction-related abilities such as distance perception and memory. We construct a diverse dataset with 330k video clips and select 2.1k high-quality samples covering varied perspectives, weather, and scenes. As existing world models differ in interaction modalities, we introduce an Action Generation Framework to unify evaluation and design six task types, generating 4.9k test samples. These tasks jointly assess model performance across visual generation, trajectory following, and memory. Evaluating 14 representative world models, we identify key limitations and provide insights for future research. The iWorld-Bench model leaderboard is publicly available at iWorld-Bench.com.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes iWorld-Bench, a benchmark for interactive world models consisting of a dataset constructed from 330k video clips with 2.1k high-quality samples selected to cover varied perspectives, weather, and scenes; a unified Action Generation Framework to handle differing interaction modalities; six task types that generate 4.9k test samples jointly assessing visual generation, trajectory following, and memory; an evaluation of 14 representative world models that identifies key limitations; and a public leaderboard at iWorld-Bench.com.

Significance. If the dataset selection and task design are shown to be representative and well-calibrated, the benchmark would address a clear gap by providing a standardized, unified evaluation protocol for physical interaction capabilities in world models, with the public leaderboard and Action Generation Framework offering concrete value for reproducibility and community progress toward more adaptive agents.

major comments (2)
  1. [Abstract and Dataset Construction] Abstract and Dataset Construction section: the claim that the 2.1k curated samples (selected from 330k clips) sufficiently represent core physical interaction capabilities rests on the statement that they cover 'varied perspectives, weather, and scenes,' but no quantitative coverage metrics, explicit selection criteria, or correlation analysis with established physical-interaction benchmarks are supplied.
  2. [Task Design and Evaluation] Task Design and Evaluation sections: the six task types are asserted to 'jointly assess' abilities and the 14-model evaluation is used to 'identify key limitations,' yet no details appear on task difficulty calibration, inter-task correlations, or statistical significance (e.g., confidence intervals or hypothesis tests) of the reported results.
minor comments (1)
  1. [Abstract] Abstract: the exact breakdown of the 4.9k test samples across the six task types is not stated, which would aid readers in assessing balance.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our manuscript. We address each major comment point by point below, indicating the revisions we will make to strengthen the paper.

read point-by-point responses

  1. Referee: [Abstract and Dataset Construction] Abstract and Dataset Construction section: the claim that the 2.1k curated samples (selected from 330k clips) sufficiently represent core physical interaction capabilities rests on the statement that they cover 'varied perspectives, weather, and scenes,' but no quantitative coverage metrics, explicit selection criteria, or correlation analysis with established physical-interaction benchmarks are supplied.

    Authors: We agree that the current description of dataset representativeness is primarily qualitative. In the revised manuscript, we will expand the Dataset Construction section with quantitative coverage metrics (e.g., distributions and percentages across perspective angles, weather categories, and scene types), explicit multi-stage selection criteria used during curation from the 330k clips, and a discussion relating these samples to core physical interaction capabilities. A direct quantitative correlation analysis with other benchmarks was not included originally due to differing evaluation protocols; we will add a qualitative comparison and flag full quantitative alignment as future work. revision: yes

  2. Referee: [Task Design and Evaluation] Task Design and Evaluation sections: the six task types are asserted to 'jointly assess' abilities and the 14-model evaluation is used to 'identify key limitations,' yet no details appear on task difficulty calibration, inter-task correlations, or statistical significance (e.g., confidence intervals or hypothesis tests) of the reported results.

    Authors: We acknowledge that additional methodological and statistical details are needed to support the claims. We will revise the Task Design and Evaluation sections to describe task difficulty calibration via pilot experiments, provide inter-task correlation analysis demonstrating complementarity across the six tasks, and report confidence intervals along with appropriate statistical tests for performance differences among the 14 models. These additions will more rigorously substantiate the joint assessment and identified limitations. revision: yes

Circularity Check

0 steps flagged

No circularity: benchmark curation and task design are explicit design choices, not derived results

full rationale

The paper is a benchmark proposal that constructs a dataset by selecting 2.1k samples from 330k clips and defines six task types plus an Action Generation Framework to unify evaluation. These steps are presented as curation and design decisions to enable testing of external world models, with no mathematical derivations, equations, fitted parameters, or predictions that reduce to the inputs by construction. No self-citations are invoked as load-bearing uniqueness theorems or ansatzes. Evaluation results on 14 representative models are external and falsifiable. The central claim therefore rests on the explicit representativeness of the curated samples rather than any internal circular reduction, making the work self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters or invented physical entities are introduced; the work relies on standard assumptions about video data representing physical interactions and the utility of curated high-quality subsets.

pith-pipeline@v0.9.0 · 5503 in / 1084 out tokens · 28856 ms · 2026-05-08T18:44:43.870930+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [1]

    Nguyen, T.-M., Yuan, S., Cao, M., Lyu, Y ., Nguyen, T

    doi: 10.1109/TIP.2012.2214050. Nguyen, T.-M., Yuan, S., Cao, M., Lyu, Y ., Nguyen, T. H., and Xie, L. Ntu viral: A visual-inertial-ranging-lidar dataset, from an aerial vehicle viewpoint.The Inter- national Journal of Robotics Research, 41(3):270–280, 2022. Patel, M., Yang, F., Qiu, Y ., Cadena, C., Scherer, S., Hutter, M., and Wang, W. Tartanground: A la...

    work page doi:10.1109/tip.2012.2214050 2012

  2. [2]

    Spatial Analysis (Indoor/Outdoor): - Determine if the agent isIndoor(enclosed) orOutdoor(open air)

  3. [3]

    - Example: ’Abandoned industrial courtyard with rusty pipes and overgrown grass’

    Detailed Scene Description: - Generate a descriptive English phrase for the specific scene visible in the frames. - Example: ’Abandoned industrial courtyard with rusty pipes and overgrown grass’

  4. [4]

    - This should be the most representative word for the place

    Scene Categorization (Dynamic Summary): - Based on your description above, summarize the scene into a SINGLE Root Noun (Category). - This should be the most representative word for the place. - Do NOT use adjectives here, just the noun

  5. [5]

    - Indoor: Lighting (Fluorescent, Dim, Natural)

    Atmospheric Analysis: - Outdoor: Weather (Sunny, Cloudy, Rainy, Night). - Indoor: Lighting (Fluorescent, Dim, Natural)

  6. [6]

    Environment_Type

    Entity Extraction: - List 15+ distinct objects visible in the scene (structural + dynamic). OUTPUT JSON FORMAT: { "Environment_Type": "Indoor" or "Outdoor", "Scene_Description": "Your detailed descriptive phrase", "Scene_Tag": "Single Root Noun (The dynamic summary)", "Weather_Lighting": "Weather or Lighting condition", "Entities": "List of objects separa...

  7. [7]

    We adopt the MUSIQ quality prediction model, leveraging its ability to perceive diverse resolutions and aspect ratios to score each frame in the video sequence

    Image Quality ( SImage).This metric assesses low-level visual distortions such as overexposure, noise, or blur in generated video frames. We adopt the MUSIQ quality prediction model, leveraging its ability to perceive diverse resolutions and aspect ratios to score each frame in the video sequence. The final score is obtained by calculating the arithmetic ...

  8. [8]

    We categorize pixel grayscales into three levels (dark, mid, bright) to construct a 3D brightness distribution vector vt = [pdark, pmid, pbright]⊤ for each frame

    Brightness Consistency (SBrightness).This metric aims to evaluate the stability of the brightness distribution in generated videos. We categorize pixel grayscales into three levels (dark, mid, bright) to construct a 3D brightness distribution vector vt = [pdark, pmid, pbright]⊤ for each frame. The comprehensive score is derived by calculating the similari...

  9. [9]

    The hue spectrum (0-179) is divided into 7 core intervals to construct a hue feature vector ht ∈R 7

    Color Temperature Constraint (SColor).To evaluate the consistency of the environmental atmosphere, we analyze the Hue dimension in the HSV color space. The hue spectrum (0-179) is divided into 7 core intervals to construct a hue feature vector ht ∈R 7. We calculate the weighted perceptual similarity of the entire sequence relative to the initial frame. To...

  10. [10]

    Sharpness Retention (SSharpness).This metric evaluates the stability of details by monitoring the evolution of edge gradients. We propose a vectorized Tenengrad method: independently calculate the sum of absolute gradients in the horizontal (Gx) and vertical (Gy) directions, and construct a 2D sharpness vector gt = (P |Gx|, P |Gy|)⊤ to 27 A Benchmark for ...

  11. [11]

    We adopt a sampling-reconstruction paradigm: discard the odd frames in the generated video and use the interpolation model to reconstruct them

    Motion Smoothness ( SMotion).This metric evaluates the sequence coherence of generated videos using the motion prior of a video interpolation model. We adopt a sampling-reconstruction paradigm: discard the odd frames in the generated video and use the interpolation model to reconstruct them. Subsequently, the smoothness is quantified by calculating the co...

  12. [12]

    The evaluation includes two stages: trajectory alignment and accuracy calculation

    Trajectory Accuracy (SAccuracy).This metric quantifies the accuracy with which the world model follows preset camera control commands. The evaluation includes two stages: trajectory alignment and accuracy calculation. First, ViPE is used to extract the original extrinsic trajectory Eraw; to eliminate coordinate system mismatch, a rotation transformation m...

  13. [13]

    Unlike STrajectory which relies on third-party estimators, this metric directly uses the system-built precise extrinsic sequence Egt as the benchmark

    Trajectory Tolerance (STolerance).This metric aims to evaluate the robustness of the model in trajectory execution under the guidance of accurate Ground-truth. Unlike STrajectory which relies on third-party estimators, this metric directly uses the system-built precise extrinsic sequence Egt as the benchmark. We adopt the same coordinate alignment and tan...

  14. [14]

    Memory Symmetry (SMemory).This metric quantifies the model’s logical loop-closure ability by checking the pixel-wise consistency of symmetric frame pairs(ft, fT−t+1 ) in cyclic or symmetric actions. We calculate the Mean Squared Error (MSEt) of symmetric frame pairs, which is then mapped to a similarity score using a compound exponential function with an ...

  15. [15]

    ViPE is used to extract the camera extrinsic parameters of each frame, resulting in a sequence {Et}T t=1, where Et ∈R 12 represents the reshaped extrinsic matrix

    Trajectory Alignment (SAlignment).This metric evaluates the model’s ability to maintain symmetric closed-loop camera trajectories in round-trip tasks. ViPE is used to extract the camera extrinsic parameters of each frame, resulting in a sequence {Et}T t=1, where Et ∈R 12 represents the reshaped extrinsic matrix. We calculate the deviation of motion featur...

    work page arXiv 2024