Pith. sign in

REVIEW 3 major objections 5 minor 46 references

A single egocentric video of one person contains enough structure to reconstruct a feasible parallel two-agent execution, and a zone-based spatial prompt lets a VLM do it with high coverage and far fewer collisions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:51 UTC pith:JMOEMBHD

load-bearing objection New problem and benchmark, but the headline collision reduction comes from ground-truth camera poses in both the prompt and the metric. the 3 major comments →

arxiv 2512.11393 v2 pith:JMOEMBHD submitted 2025-12-12 cs.CV

The N-Body Problem: Parallel Execution from Single-Person Egocentric Video

Zhifan Zhu , Yifei Huang , Yoichi Sato , Dima Damen This is my paper
classification cs.CV
keywords N-Body Problemegocentric videoparallel task executionvision-language modelsspatial reasoningaction coveragecollision avoidancetask scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces the N-Body Problem: given one egocentric video of a single person performing a long, unscripted activity, predict how N hypothetical agents could perform the same work in parallel to finish faster. A valid parallel execution must respect real-world feasibility: agents cannot occupy the same space, use the same object simultaneously, or violate causal prerequisites. The authors formalize this with a suite of metrics for goals (coverage, speed-up) and constraints (collision, jump, object conflict, causality violation). They show that Gemini 2.5 Pro, guided by a structured prompt that discretizes the 3D environment into zones and lists when the camera wearer is in each zone, raises action coverage from 62.9% to 91.3% while cutting collision, object, and causality violations by roughly half on HD-EPIC. The paper argues the problem is tractable with current VLMs plus structured spatial prompting, and that the proposed metrics provide a way to evaluate such parallel plans.

Core claim

The central claim is that a single-person egocentric video contains enough information to reconstruct a causally and physically plausible parallel execution by N agents, and that the main bottleneck is spatial reasoning: raw VLMs produce plans with high collision rates. The paper's discovery is that encoding the camera wearer's trajectory as a discretized zone schedule—a CSV of (start-time, end-time, zone) triples derived from ground-truth camera poses—and instructing the model to avoid assigning two agents to the same zone concurrently reduces collision rate from 17.2% to 7.7% on HD-EPIC while increasing action coverage to 91.3%. This spatial prompt is the load-bearing component; without it

What carries the argument

The spatial prompt: the egocentric trajectory is divided into equal-sized zones (120x120 cm) on the ground plane, and the video is converted into a list of (start-time, end-time, zone) triplets. The VLM is told to rely exclusively on this CSV when reasoning about space and to serialize any segments that fall in the same zone. This converts a continuous 3D collision-avoidance problem into a discrete temporal constraint that a language model can follow. The evaluation suite is the other key piece: ground-truth camera poses, action annotations, object tracks, and recipe dependencies from HD-EPIC and EPIC-KITCHENS are used to score collision, object conflict, and causality violation rates.

Load-bearing premise

The spatial prompt is built from ground-truth camera poses, and the same poses are used to measure collision; if poses are missing or noisy, the zone schedule would be wrong and the collision reduction would likely vanish.

What would settle it

Take a held-out set of egocentric videos, corrupt or drop up to 50% of the camera poses when building the zone CSV, then measure the collision rate of the generated plan; if the collision rate rises back to the ~17% baseline, the spatial-prompt effect depends on perfect pose input. Alternatively, run the same method on videos where two agents can legitimately coexist in one zone (e.g., a wide kitchen island) and check whether over-serialization artificially caps speed-up.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the spatial-prompt approach holds, long egocentric video can serve as a low-cost source of multi-agent execution plans: watch one person work, then produce a two- or three-agent schedule with 1.4–1.6x speed-up and single-digit collision rates.
  • The metric suite defines a standard way to evaluate parallel-execution plans without a single ground-truth plan, since multiple valid executions exist for the same activity.
  • The zone-granularity trade-off (larger zones → lower collision but less speed-up) gives practitioners a dial to control the safety/efficiency balance of generated plans.
  • Because naive half-half splitting and a HEFT-style scheduler both yield high collision rates, the paper suggests that constraint-aware reasoning in a VLM, not just optimization, is needed for this task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same zone-prompt trick could be extended to other spatial-reasoning tasks in long video, such as predicting dual-robot manipulation or safety-checking human activity plans, wherever camera poses or trajectories are available.
  • If camera poses are noisy or absent at inference, the collision-reduction benefit likely disappears; the paper's reliance on ground-truth poses for both prompt construction and evaluation is the main thing to test next.
  • The model's failure to infer causal prerequisites like 'grinding coffee precedes brewing' suggests a natural next benchmark: teaching VLMs to discover unseen dependencies from object-state changes rather than from recipe annotations.
  • One could test whether a smaller open-weight VLM, given the same zone CSV and a few-shot example, can approach Gemini's coverage; the paper shows Qwen2.5-VL-72B currently fails to produce output on many videos.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the N-Body Problem: given a single-person egocentric video, predict how N hypothetical agents could perform the observed activities in parallel. The authors formalize input/output representations, define goal metrics (coverage, speed-up) and constraint metrics (spatial collision, spatial jump, object conflict, causality violation), and propose a prompting strategy for Gemini 2.5 Pro. The final prompt includes a spatial prompt derived from a zone-time CSV that encodes the source person's trajectory. On 100 long videos from HD-EPIC and EPIC-KITCHENS, the approach improves action coverage over a base prompt (e.g., 62.9% → 91.3% on HD-EPIC for N=2) while reducing collision, object conflict, and causality violation rates. The paper also compares against naive half-half, HEFT-style schedulers, and Qwen2.5-VL-72B.

Significance. The problem formulation and the proposed metric suite are novel and potentially useful: they address a realistic and under-explored question of re-scheduling single-agent video content into multi-agent parallel plans, with explicit physical and semantic feasibility constraints. The analysis of VLM behavior under progressively richer prompts is also informative. However, the central claim that the method predicts parallel execution from a single egocentric video is compromised by the spatial prompt: it is built from ground-truth camera poses that are also used to compute the collision rate. As a result, the reported collision reduction largely reflects the model following an externally supplied privileged schedule rather than spatial reasoning from the video. This issue is load-bearing and must be addressed before the results can be interpreted as claimed.

major comments (3)
  1. [Sec. 4.1, Sec. 4.3, Supp. D] The spatial prompt is constructed from ground-truth camera poses. Sec. 4.1 states that camera poses provide the person's trajectory and are used to evaluate Collision Rate, and Sec. 4.3 describes extracting zone durations from that trajectory into a CSV. This directly contradicts the statement in Sec. 4.1 that 'this ground truth is not used as input.' Moreover, Supp. D instructs the model: 'The spatial reasoning should rely on the trajectory csv exclusively, do not use any knowledge from the video.' Thus the collision-rate drop (17.5% → 7.7% in Table 1) is not evidence of egocentric spatial reasoning; it is the expected consequence of telling the model to avoid concurrent occupancy of zones derived from the same GT poses used in Eq. (14). To support the paper's core claim, the spatial prompt must be built from video-only pose estimation (e.g., monocular SLAM or COLMAP on the egocentric v
  2. [Sec. 3.2 vs. Sec. 4.2, Eq. (6) and Eq. (13)] The speed-up definition is internally inconsistent. Eq. (6) defines Speed-Up as 'Sequential execution time / Parallel execution time', but Eq. (13) implements the numerator as the number of original frames covered by assigned segments, not the full video duration. A model can therefore increase its reported speed-up simply by discarding uncovered frames. Since coverage and speed-up are reported separately this is partly mitigated, but the metric as defined can be gamed and does not match the stated formal definition. Please use the full sequential duration in the numerator, or explicitly define a 'covered speed-up' and justify why dropping work should improve it.
  3. [Tables 1 and 2, Qwen comparison] The open-weight VLM Qwen2.5-VL-72B produces outputs on only 51/80 HD-EPIC and 7/20 EPIC videos; the tables report averages over these subsets while Gemini results are over the full sets. The statement that Qwen 'fails to speed-up' is therefore based on non-comparable subsets. Please report results on the common subset of videos where Qwen succeeded, or clearly break down results by output availability, so that the comparison is meaningful.
minor comments (5)
  1. [Supp. D, line 15] Typo: 'reasong' should be 'reasoning'.
  2. [Supp. B, paragraph 2] The sentence 'Increasing zone sizes leads to more reduced collision rates, but the speed-up increases accordingly, i.e. you are speeding-up less' is contradictory; the intended meaning appears to be that speed-up decreases (you speed up less) as zone size grows. Please rephrase.
  3. [Fig. 1] The figure caption and main text report a 1.6x speed-up for the 2-body case, but the figure includes '21.6x'. Please correct this inconsistency.
  4. [Sec. 3.1, Eq. (3)] The notation P_n[τ, τ+(j−i)] = S_ij may be confusing because S_ij is defined by its start and end indices, not its duration. Define the segment length explicitly or use |S_ij| consistently.
  5. [Sec. 4.2, Eq. (17)] The causality violation definition in Eq. (17) checks E(G_l,0) > S(G_l,1), but the text says a violation also occurs if the prerequisite segment is missing. Please specify how missing segments are handled in the implementation.

Circularity Check

1 steps flagged

Spatial prompt is built from ground-truth camera poses that also define the collision metric, so the reported collision reduction is partly forced by the input rather than demonstrated spatial reasoning from video.

specific steps
  1. self definitional [Sec. 4.1 / Sec. 4.3 / Supp. D; Eq. (14); Table 1]
    "Crucially, this ground truth is not used as input, but only to evaluate the feasibility of proposed executions. ... With the person’s trajectory given by the camera poses in the source video, we are able to evaluate the Collision Rate. ... We then extract the duration when the person in I remains within one zone, producing a list of triplets of: (start-time, end-time, zone number). We then instruct Gemini 2.5 Pro to avoid assigning two parallel agents working in the same zone concurrently. ... The spatial reasong should rely on the trajectory csv exclusively, do not use any knowledge from the"

    The zone-time CSV supplied in the spatial prompt is derived from the same ground-truth camera trajectory that is used to compute the collision-rate metric (Eq. 14, via is_collide on Γ_n). The prompt instructs the VLM to assign agents to different zones and to rely on the CSV exclusively. Thus the reported collision-rate drop (17.5%→7.7% in Table 1) is largely a direct consequence of providing the evaluator's own ground-truth signal as input, rather than evidence that the VLM reasons about 3D space from egocentric video. The paper states 'this ground truth is not used as input', but for camera poses the opposite is true: the spatial prompt is a discretized version of those poses. The collision-reduction claim therefore reduces by construction to the input; coverage and OCR/CVR improvements

full rationale

The central circularity is the collision-rate result. The spatial prompt is constructed from ground-truth camera poses (via zone occupancy), and the collision metric is computed from the same ground-truth poses. The prompt explicitly tells the VLM to rely on the trajectory CSV and not the video, so the measured collision reduction is not a test of video-derived spatial reasoning but of following a privileged schedule. This makes the headline collision improvement partly circular. However, the coverage improvement (guided by goals in the prompt) and the object-conflict/causality reductions are not defined in terms of the same input signal and retain independent content. No load-bearing self-citation chain or imported uniqueness theorem is present; the use of HD-EPIC by the authors is as an external benchmark. Overall, partial circularity affecting one of the three headline constraint metrics, so score 6.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central method relies on ground-truth camera poses as input for the spatial prompt and as ground truth for the collision metric, which is a strong domain assumption. The manual causal annotations and body-size model are additional ad-hoc choices. No new physical entities are introduced.

free parameters (2)
  • Zone size (120x120 cm) = 120x120 cm
    Chosen after ablation of 40x40, 80x80, GMM variants; larger zones reduce collision but also reduce speed-up (Table 3).
  • Human body dimensions for collision (146 cm wide x 25 cm deep) = 146 cm x 25 cm
    Used in the is_collide function to determine spatial occupation from camera poses (Sec 4.2).
axioms (4)
  • domain assumption Ground-truth camera poses (SLAM/COLMAP) accurately represent person locations
    Used to build the spatial zone CSV and to evaluate collision rate (Sec 4.1, 4.2, 4.3).
  • domain assumption Object exclusivity: no additional object copies exist
    Stated in Sec 3.3; OCR assumes using the same object instance by two agents is always a conflict.
  • ad hoc to paper Manual recipe step-step dependencies are correct
    Authors manually annotated 204 step-step dependencies across 56 recipes without inter-annotator agreement or external validation (Supp C).
  • domain assumption Collision can be measured by projecting camera pose with a fixed body bounding box
    The is_collide function uses a hand-set body size to decide if two agents occupy the same space (Sec 4.2).

pith-pipeline@v1.3.0-alltime-deepseek · 15876 in / 8211 out tokens · 72116 ms · 2026-08-03T16:51:53.524960+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of The N-Body Problem: Parallel Execution from Single-Person Egocentric Video." pith.science (2026). https://pith.science/paper/JMOEMBHD

@misc{pith2026251211393,
  author       = {Pith},
  title        = {Pith review of: The N-Body Problem: Parallel Execution from Single-Person Egocentric Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JMOEMBHD}},
  note         = {Machine review of arXiv:2512.11393}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Humans can intuitively parallelise complex activities, but can a model predict this from observing a single person? Given one egocentric video, we introduce the N-Body Problem: predicting how N individuals, can hypothetically perform the same set of tasks. The goal is to maximise speed-up, but naive assignment of video segments to individuals often violates real-world constraints, leading to physically impossible scenarios like two people using the same object or occupying the same space. To quantify this, we formalise the N-Body Problem and propose a suite of metrics to evaluate both performance (speed-up, task coverage) and feasibility (spatial collisions, object conflicts and causal constraints). As a proof of concept, we introduce a structured prompting strategy that guides a Vision-Language Model (VLM) to reason about the 3D environment, object usage, and temporal dependencies, producing a viable parallel execution. On 100 videos from EPIC-Kitchens and HD-EPIC, for $N = 2$, our structured prompt improves action coverage by 45% over a baseline prompt for Gemini 2.5 Pro, while simultaneously slashing collision rates, object and causal conflicts by 51%, 52% and 55% respectively.

Figures

Figures reproduced from arXiv: 2512.11393 by Dima Damen, Yifei Huang, Yoichi Sato, Zhifan Zhu.

Figure 1
Figure 1. Figure 1: Top. Our input is single-person egocentric video. The camera wearer performs a combination of cooking, washing up and ordering. Middle. Predicted 2-Body Parallel execution. Our method, which uses Gemini with prompts on goals and constraints, achieves a 1.6x speed-up (from 19.8min to 10.4min) with 86% coverage. We show 3D representation of where the 2-agents (orange/blue) are and their camera views. P1 is m… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of evaluation metrics across all 100 videos [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative results. Top: Same video with N = 2 and N = 3. Left: P1 is marinating chicken and washing up. P2 gathers spices (for the marination) in advance then prepares eggs. Right: P2 mostly fetches items from around the kitchen. Bottom Left: (N = 2) P2 is left to do the washing up and clearing. Bottom Right: (N = 3) P1 is cooking, P2 is pouring a glass of wine then puts stuff away. P3 is emptying the di… view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of parallel executions. Left: Distribution of agent cooking time (normalised by agent total time). Top example: P1 mostly cooks while P2 does the cleaning. Bottom: P1 and P2 both clean. Middle: The ratio of prep carried out by the same agent (Self) vs by the Other agent. Left example: P2 fetches the eggs and the bowl, P1 breaks the egg. Right: P2 fills the mixer cup and starts mixing Right: Distri… view at source ↗
Figure 5
Figure 5. Figure 5: See also Video Supp: Top. Our input is single-person egocentric video. The camera wearer (shown also in orange in 3D) performs a combination of cooking, washing up and ordering. Middle. 2-Body Parallel execution. Our method achieves a 1.6x speed-up (from 19.8min to 10.4min) with 86% coverage. We show coloured segments (orange/blue) along with 3D representation of where the 2-agents are and their camera vie… view at source ↗
Figure 6
Figure 6. Figure 6: Recipe order distribution over methods. Method Coverage (%)↑ Action Coverage (%) ↑ Speed-Up ↑ Collision Rate (%) ↓ Avg. Jump (m) ↓ OCR (%) ↓ CVR (%) ↓ +Goals-and-Constraints 87.4 88.1 1.59 17.5 0.47 0.93 26.3 Raw Trajectory 90.6 91.3 1.49 13.9 0.48 1.17 15.7 GMM (5 comps) 91.3 91.4 1.48 10.8 0.57 0.57 20.3 GMM (10 comps) 86.8 87.0 1.44 10.5 0.55 0.89 21.4 40 × 40cm 88.3 88.8 1.48 11.5 0.48 0.90 19.8 80 × 8… view at source ↗
Figure 8
Figure 8. Figure 8: Speed-Up and Collision trade-off of different methods [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Step-step causal dependency annotation samples. We [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 8 linked inside Pith

  1. [1]

    Video-mined task graphs for keystep recognition in instructional videos.Ad- vances in Neural Information Processing Systems, 2023

    Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Tri- antafyllos Afouras, and Kristen Grauman. Video-mined task graphs for keystep recognition in instructional videos.Ad- vances in Neural Information Processing Systems, 2023. 1

    2023

  2. [2]

    SpatialVLM: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endow- ing vision-language models with spatial reasoning capabili- ties. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024. 3

    2024

  3. [3]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 2, 6

    Pith/arXiv arXiv 2025

  4. [4]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.International Journal of Computer Vision, 2022

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.International Journal of Computer Vision, 2022. 1, 2, 3, 4

    2022

  5. [5]

    Scheduling parallel tasks: Approximation algorithms

    Pierre-Franc ¸ois Dutot, Gr ´egory Mouni ´e, and Denis Trys- tram. Scheduling parallel tasks: Approximation algorithms. Handbook of scheduling: Algorithms, models, and perfor- mance analysis, 2004. 3

    2004

  6. [6]

    Project aria: A new tool for egocentric multi-modal ai research.arXiv preprint arXiv:2308.13561, 2023

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research.arXiv preprint arXiv:2308.13561, 2023. 5

    Pith/arXiv arXiv 2023

  7. [7]

    Anticipative video transformer

    Rohit Girdhar and Kristen Grauman. Anticipative video transformer. InProceedings of the IEEE International Con- ference on Computer Vision, 2021. 3

    2021

  8. [8]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 3

    2022

  9. [9]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InProceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, 2024. 1, 3

    2024

  10. [10]

    EgoExoBench: A benchmark for first-and third-person view video understanding in mllms

    Yuping He, Yifei Huang, Guo Chen, Baoqi Pei, Jilan Xu, Tong Lu, and Jiangmiao Pang. EgoExoBench: A benchmark for first-and third-person view video understanding in mllms. arXiv preprint arXiv:2507.18342, 2025. 3

    Pith/arXiv arXiv 2025

  11. [11]

    Predicting gaze in egocentric video by learning task- dependent attention transition

    Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. Predicting gaze in egocentric video by learning task- dependent attention transition. InProceedings of the Euro- pean Conference on Computer Vision, 2018. 3

    2018

  12. [12]

    Improving action segmentation via graph-based temporal reasoning

    Yifei Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph-based temporal reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020. 3

    2020

  13. [13]

    EgoExoLearn: A dataset for bridging asynchronous ego- and exo-centric view of proce- dural activities in real world

    Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Li- jin Yang, Baoqi Pei, Hongjie Zhang, Dong Lu, Yali Wang, Limin Wang, and Yu Qiao. EgoExoLearn: A dataset for bridging asynchronous ego- and exo-centric view of proce- dural activities in real world. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

  14. [14]

    Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities

    Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Zhu. Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. InProceedings of the European Conference on Computer Vision. Springer, 2020. 3

    2020

  15. [15]

    EgoTaskQA: Understanding human tasks in ego- centric videos.Advances in Neural Information Processing Systems, 2022

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. EgoTaskQA: Understanding human tasks in ego- centric videos.Advances in Neural Information Processing Systems, 2022. 3

    2022

  16. [16]

    Open-ended hierarchical streaming video understanding with vision language models

    Hyolim Kang, Yunsu Park, Youngbeom Yoo, Yeeun Choi, and Seon Joo Kim. Open-ended hierarchical streaming video understanding with vision language models. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), 2025. 3

    2025

  17. [17]

    Epic-fusion: Audio-visual temporal binding for egocentric action recognition

    Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. InProceedings of the IEEE International Conference on Computer Vision, 2019. 3

    2019

  18. [18]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Zhongcong Xu, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, W ANG HongFa, Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretraining. InAdvances in Neural Information Processing Systems, 2022. 3

    2022

  19. [19]

    3DAxisPrompt: Promoting the 3D grounding and reasoning in GPT-4o.Neu- rocomputing, 2025

    Dingning Liu, Cheng Wang, Peng Gao, Renrui Zhang, Xinzhu Ma, Yuan Meng, and Zhihui Wang. 3DAxisPrompt: Promoting the 3D grounding and reasoning in GPT-4o.Neu- rocomputing, 2025. 3

    2025

  20. [20]

    Action dynamics task graphs for learning plannable representations of procedural tasks.arXiv preprint arXiv:2302.05330, 2023

    Weichao Mao, Ruta Desai, Michael Louis Iuzzolino, and Nitin Kamra. Action dynamics task graphs for learning plannable representations of procedural tasks.arXiv preprint arXiv:2302.05330, 2023. 1, 3

    Pith/arXiv arXiv 2023

  21. [21]

    Visual agentic AI for spatial reasoning with 9 a dynamic api

    Damiano Marsili, Rohun Agrawal, Yisong Yue, and Geor- gia Gkioxari. Visual agentic AI for spatial reasoning with 9 a dynamic api. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025. 3

    2025

  22. [22]

    Task switching.Trends in cognitive sci- ences, 7(3):134–140, 2003

    Stephen Monsell. Task switching.Trends in cognitive sci- ences, 7(3):134–140, 2003. 1

    2003

  23. [23]

    Spare: Enhancing spatial rea- soning in vision-language models with synthetic data.arXiv preprint arXiv:2504.20648, 2025

    Michael Ogezi and Freda Shi. Spare: Enhancing spatial rea- soning in vision-language models with synthetic data.arXiv preprint arXiv:2504.20648, 2025. 3

    Pith/arXiv arXiv 2025

  24. [24]

    HD-EPIC: A Highly-Detailed Egocentric Video Dataset

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, Jacob Chalk, Zhi- fan Zhu, Rhodri Guerrier, Fahd Abdelazim, Bin Zhu, Da- vide Moltisanti, Michael Wray, Hazel Doughty, and Dima Damen. HD-EPIC: A Highly-Detailed Egocentric Video Dataset. InProceedings of the...

    2025

  25. [25]

    Spatial cognition from egocentric video: Out of sight, not out of mind

    Chiara Plizzari, Shubham Goel, Toby Perrett, Jacob Chalk, Angjoo Kanazawa, and Dima Damen. Spatial cognition from egocentric video: Out of sight, not out of mind. In2025 International Conference on 3D Vision (3DV), 2025. 3

    2025

  26. [26]

    Unweavenet: Unweaving activity stories

    Will Price, Carl V ondrick, and Dima Damen. Unweavenet: Unweaving activity stories. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,

  27. [27]

    Join me: Empirical means for gathering col- laborative affordances

    Rachel Ringe, Mihai Pomarlan, Robert Porzel, and Rainer Malaka. Join me: Empirical means for gathering col- laborative affordances. InMensch und Computer 2025- Workshopband, 2025. 3

    2025

  28. [28]

    Executive control of cognitive processes in task switching

    Joshua S Rubinstein, David E Meyer, and Jeffrey E Evans. Executive control of cognitive processes in task switching. Journal of experimental psychology: human perception and performance, 2001. 1

    2001

  29. [29]

    Understanding multi-task activities from single-task videos

    Yuhan Shen and Ehsan Elhamifar. Understanding multi-task activities from single-task videos. InProceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, 2025. 1, 3

    2025

  30. [30]

    Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics

    Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025. 3

    2025

  31. [31]

    Multi-agent path finding–an overview.Artificial intelligence: 5th RAAI summer school, 2019

    Roni Stern. Multi-agent path finding–an overview.Artificial intelligence: 5th RAAI summer school, 2019. 3

    2019

  32. [32]

    Mind the gap: Benchmarking spatial reasoning in vision-language models.arXiv preprint arXiv:2503.19707,

    Ilias Stogiannidis, Steven McDonagh, and Sotirios A Tsaf- taris. Mind the gap: Benchmarking spatial reasoning in vision-language models.arXiv preprint arXiv:2503.19707,

    Pith/arXiv arXiv

  33. [33]

    Performance-effective and low-complexity task scheduling for heterogeneous computing.IEEE transactions on parallel and distributed systems, 2002

    Haluk Topcuoglu, Salim Hariri, and Min-You Wu. Performance-effective and low-complexity task scheduling for heterogeneous computing.IEEE transactions on parallel and distributed systems, 2002. 3, 6

    2002

  34. [34]

    Epic fields: Marrying 3d geometry and video under- standing.Advances in Neural Information Processing Sys- tems, 2023

    Vadim Tschernezki, Ahmad Darkhalil, Zhifan Zhu, David Fouhey, Iro Laina, Diane Larlus, Dima Damen, and Andrea Vedaldi. Epic fields: Marrying 3d geometry and video under- standing.Advances in Neural Information Processing Sys- tems, 2023. 5

    2023

  35. [35]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 2024

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.Advances in Neural Information Processing Systems, 2024. 3

    2024

  36. [36]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 6

    Pith/arXiv arXiv 2024

  37. [37]

    Retrieval-augmented egocentric video captioning

    Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval-augmented egocentric video captioning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024. 3

    2024

  38. [38]

    Perceiving and acting in first-person: A dataset and benchmark for egocentric human-object-human interac- tions.arXiv preprint arXiv:2508.04681, 2025

    Liang Xu, Chengqun Yang, Zili Lin, Fei Xu, Yifan Liu, Con- gsheng Xu, Yiyi Zhang, Jie Qin, Xingdong Sheng, Yunhui Liu, et al. Perceiving and acting in first-person: A dataset and benchmark for egocentric human-object-human interac- tions.arXiv preprint arXiv:2508.04681, 2025. 3

    Pith/arXiv arXiv 2025

  39. [39]

    How to enable LLM with 3D capacity? a survey of spatial reasoning in llm.arXiv preprint arXiv:2504.05786,

    Jirong Zha, Yuxuan Fan, Xiao Yang, Chen Gao, and Xinlei Chen. How to enable LLM with 3D capacity? a survey of spatial reasoning in llm.arXiv preprint arXiv:2504.05786,

    arXiv

  40. [40]

    Masked video and body-worn imu autoencoder for egocentric action recognition

    Mingfang Zhang, Yifei Huang, Ruicong Liu, and Yoichi Sato. Masked video and body-worn imu autoencoder for egocentric action recognition. InProceedings of the Euro- pean Conference on Computer Vision. Springer, 2024. 3

    2024

  41. [41]

    + Goal-Only

    Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, and Angela Yao. Egotextvqa: Towards egocentric scene-text aware video question answering. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2025. 3 10 The N-Body Problem: Parallel Execution from Single-Person Egocentric Video Supplem...

    2025

  42. [211]

    When making coffee, pour hot water into the coffee before the coffee powder has been added to the mug

  43. [222]

    Wash mixer’s head while the other agent is still using the mixer

  44. [233]

    P1" and

    Uses the same trash bin at the same time. 24 25# Covering every moment in the original video 26 27Note that although this task aims to speed up the video, it should not skip any part of the original video. Every second of the original video needs to be covered, ensuring coverage = 100%. 28 29# Format 30 31Output json format: 32With two keys "P1" and "P2",...

  45. [921]

    For example, no simultaneous access to the fridge

    Pay attention to Spatial Conflicts. For example, no simultaneous access to the fridge. A better planning may use the same countertop at different times

  46. [932]

    place mug before pouring milk

    Pay attention to the Object dependency, e.g. place mug before pouring milk. 16