{"id":"246f8d1d-8532-4500-91c1-119961d01977","arxiv_id":"2412.00932","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FICTION predicts future 3D interaction locations and body poses up to three minutes ahead from egocentric video and a 3D scene map, and claims substantial gains over prior methods on a new Ego-Exo4D benchmark.","lead":"This paper introduces FICTION, a model that watches egocentric video and predicts, in 3D space and over the next three minutes, which objects a person will touch and the body pose they will use. It also builds a new benchmark from Ego-Exo4D and reports large gains over prior 2D and autoregressive methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Object map S is not temporally constrained to frames up to tau_o; Appendix E's static-point-cloud admission indicates it uses full-take data, so reported gains may be inflated by future scene leakage.","rationale":"The reader's weakest assumption is exactly the most load-bearing concern: the temporal construction of the 3D object map. I agree with that assessment. The concern is not merely a missing detail; Appendix E's statement that the dataset assumes a static point cloud and explicitly chooses not to use only the last time segment strongly suggests S includes object placements observed after tau_o. This matters because the task is defined as predicting interactions given observation up to tau_o (Eq. 1), and because the environment map is a primary input. If S contains future object placements, the model is given information unavailable to a system that only watches the video up to tau_o, and the comparison to baselines without such a map is unfair. The central claim would survive if S were demonstrated to be a prior environment map available at tau_o, or if re-running with a temporally restricted S preserved the reported gains. The abstract's 'more than 30% relative gains' overstatement is real but secondary; the temporal-leakage issue affects the validity of the comparison rather than just its magnitude. For that reason I keep the reader's CONDITIONAL verdict: the paper should be accepted only after the authors clarify the temporal construction of S and either justify it as a prior map or re-run the experiments with a cut-off at tau_o. No change to the reader's verdict is needed.","tokens_in":20972,"tokens_out":7383,"duration_ms":71641,"concrete_test":"In the dataset-construction code (or by re-running the pipeline), record for each object bounding box in S the earliest video-frame timestamp of any Detic detection contributing to it. If any box has earliest timestamp > tau_o (30s), future frames leak into S. Then rebuild S using only frames with timestamp <= tau_o (or from an explicitly independent prior scan) and re-run the Table 1 location-prediction evaluation. If PR-AUC drops by more than about 2 points on any scenario, or the relative gain over the best baseline shrinks materially, the reported gains are partly an artifact of future scene leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that FICTION predicts future interactions from an observation up to time tau_o (Sec. 3.1, Eq. 1). The 3D object placement map S is a main input (Sec. 3.2, 'Encoding object bounding boxes'), but the paper never states that S is built only from frames with timestamps <= tau_o. In Sec. 3.3, S is constructed by running Detic on video frames at 30 fps and clustering the resulting 3D points; no temporal cutoff is specified. Appendix E ('Limitations', supplementary p. 3) explicitly says: 'We also assume a static point cloud when creating the dataset, while in practice, the object location can change with time. It is possible to use 3D information only from the last time segment for improving the spatial input to the model, we do not consider this case for the ease of the I/O.' This indicates S aggregates object placements across the full take, including times after tau_o. If an object is first detected after tau_o but is present in S, the model knows which objects exist in the environment and where, leaking future scene content into the 'observation.' The ablations show how much the model depends on S: without environment context, PR-AUC drops from 21.0 to 9.9 on cooking, 18.7 to 6.0 on bike repair, and 12.7 to 4.7 on health. Baselines do not receive this map. The comparison is therefore not a like-for-like test of prediction from past video unless S comes from a genuinely prior environment scan, which the paper does not establish.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FICTION, a method and benchmark for 4D future interaction prediction from egocentric video. Given an observation window up to time tau_o, the model predicts (i) all future 3D object interaction locations within a tau_f = 3-minute horizon, encoded as a voxel grid, and (ii) a distribution of SMPL body poses at each query interaction location, modeled with a CVAE. The model fuses three inputs: egocentric video features (EgoVLPv2), the actor's observed body pose (WHAM), and a voxelized 3D object placement map S built from Detic detections and DBSCAN clustering. The dataset is curated from Ego-Exo4D procedural videos (cooking, bike repair, health) with interaction instances defined by hand-in-3D-bounding-box plus Llama-3.1 narration matching. Experiments compare against six baselines (HierVL, OCT, OccFormer, VoxFormer, 4D-Humans, T2M-GPT, and a video-to-pose CVAE) and report PR-AUC and Chamfer distance for locations and MPJPE/PA-MPJPE for poses, showing consistent improvements on all three scenarios.","tokens_in":21305,"tokens_out":4352,"duration_ms":38823,"significance":"If the empirical claims hold, this is a novel and worthwhile task formulation that connects long-horizon activity anticipation with explicit 3D environment context, and the released dataset could enable follow-up work. The architecture is simple and interpretable, and the paper includes useful ablations (video, pose, environment) and hyperparameter studies in the supplementary material. However, the central comparison is compromised by an unspecified temporal cutoff in the construction of the input object map S, and the headline '>30% relative gains' is not supported by the paper's own tables. The significance of the contribution is therefore contingent on resolving these issues.","major_comments":[{"comment":"The 3D object placement map S, a primary input, is not explicitly constrained to frames with timestamps up to tau_o. Section 3.2 describes building S as a voxel grid with object indices and the actor location, and Section 3.3 says object bounding boxes are computed by running Detic at 30 fps on video frames and clustering the resulting 3D points, without any temporal cutoff. Appendix E states: 'We also assume a static point cloud when creating the dataset... It is possible to use 3D information only from the last time segment for improving the spatial input to the model, we do not consider this case for the ease of the I/O.' This strongly suggests that S aggregates object placements across the full take, including frames after tau_o. If an object first appears after the observation window but is present in S, the model knows that the object exists and where it is, leaking future scene content into the 'observation.' The ablations show the model depends heavily on S (w/o env PR-AUC drops from 21.0 to 9.9 on cooking, 18.7 to 6.0 on bike repair, and 12.7 to 4.7 on health), and baselines do not receive S. The reported gains may therefore be inflated by this leakage. The manuscript must either state that S is built only from frames up to tau_o, or clarify that S comes from a genuinely prior environment scan independent of the current take; in the latter case, baselines should be given equivalent information.","section":"Sec. 3.2, Sec. 3.3, Appendix E"},{"comment":"The abstract and introduction claim 'more than 30% relative gains' over the best baseline, but Table 1 does not support this as a summary statement. Location PR-AUC relative improvements are 24.3% on cooking (21.0 vs 16.9), 32.6% on bike repair (18.7 vs 14.1), and 13.4% on health (12.7 vs 11.2). Pose MPJPE improvements are 13.3% on cooking (229 vs 264), 7.5% on bike repair (372 vs 402), and 22.2% on health (172 vs 221). Only the bike-repair location setting exceeds 30%. The claim should be qualified, e.g., 'up to 32% relative gain,' or the precise settings should be named.","section":"Abstract, Sec. 1, Table 1"},{"comment":"No error bars, standard deviations, confidence intervals, or significance tests are reported for any metric. The text repeatedly states that FICTION 'significantly outperforms' all baselines, but the margin on health location PR-AUC is only 1.5 points (12.7 vs 11.2), and on pose PA-MPJPE the differences are 4-6 mm. Given that the ground-truth interactions are produced by an automatic pipeline (Detic, WHAM, Llama-3.1), a few runs with different seeds or a bootstrap over test episodes would be needed to establish that these gaps are not noise.","section":"Sec. 4, Table 1"},{"comment":"The pose distribution is evaluated by sampling N=5 poses and selecting the one closest to ground truth (MPJPE and PA-MPJPE). This best-of-N protocol rewards diversity without penalizing implausible samples, so the reported numbers can be artificially low if the model produces a broad distribution. The paper should also report the average error over all samples, or a coverage/frequency metric over the interaction locations, so the reader can assess the quality of the full distribution rather than only the closest sample.","section":"Sec. 3.3, Sec. 4 (metrics)"},{"comment":"The dataset curation pipeline defines an interaction as hands inside an object's 3D bounding box plus an LLM-based narration match. The paper does not report the precision/recall of this automatic pipeline against a human-annotated subset, nor the rate of agreement between the geometric cue and the narration cue. Since the benchmark is being released, a validation of the interaction annotation quality is important for its credibility and for interpreting the results.","section":"Sec. 3.3, Appendix B"}],"minor_comments":[{"comment":"The sentence 'Note that this simplification does not affect the curated dataset quality, since we use narrations from Ego-Exo4D as an additional signal' is a non-sequitur: the static point-cloud assumption is about the model input, not about dataset quality, and the narrations do not compensate for future object locations being visible in S.","section":"Supplementary Appendix E"},{"comment":"There is a typo: 'we do not over-emphaisze on the hands' should be 'emphasize.'","section":"Supplementary Appendix B"},{"comment":"Inconsistent spacing appears in the model name ('FI CTION' in section headings and Figure 2) and in baseline names ('V oxFormer' in the baselines paragraph and 'CV AE' in Section 3.2); consider unifying these.","section":"Throughout"},{"comment":"Equation (1) defines Fo as returning a set of points x in R^3, but Section 3.2 and the loss describe a binary voxel grid output; please align the notation so it is clear whether the output is a set of points or a voxel map.","section":"Sec. 3.1, Eq. (1)"},{"comment":"Figure 2 does not indicate the temporal cutoff tau_o in the observation inputs, nor does it show which inputs are used for the location decoder versus the CVAE; adding these annotations would improve readability and prevent the temporal-cutoff ambiguity from persisting in the figure.","section":"Sec. 3.2, Figure 2"},{"comment":"The paper does not specify how many random seeds were used for training or whether the reported results are averaged over seeds; please state this for all models.","section":"Sec. 3.4, Sec. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a strong group and the dataset release is potentially valuable, but the temporal leakage concern in the construction of S is a load-bearing issue that could undermine the core comparison if not addressed. I would encourage the editor to ask for a clear statement of the temporal window used to build S, and to ensure the revised version also corrects the unsupported '>30%' claim and reports variance estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the FICTION paper. The core idea is genuinely new: no one has formulated anticipation as predicting 3D interaction locations plus a distribution of body poses at those locations over a multi-minute horizon. The Ego-Exo4D-derived benchmark is a useful artifact, and the authors promise code and data. The model itself is not exotic—multimodal transformer + CVAE—but it is a sensible baseline for the task. The ablations show the environment context is doing most of the work, which is an interesting finding in itself.\n\nThe soft spots are real. The biggest one is the temporal construction of the 3D object map S. The paper never states that S is built only from frames up to tau_o. Appendix E says they 'assume a static point cloud' and that using '3D information only from the last time segment' is something they do not consider. That reads like S aggregates object placements across the entire take. If so, the model knows which objects exist in the environment and where, including objects the person has not yet seen at observation time. Baselines do not receive this map. The ablation without env context drops PR-AUC from 21.0 to 9.9, so the map is doing enormous work. That makes the headline comparison structurally unfair unless S is either restricted to observed frames or also provided to baselines. This is not a hook to reject, but it has to be fixed or argued as a legitimate prior map.\n\nNo error bars anywhere, and the abstract's 'more than 30% relative gains' is not supported across the board: location gains range from 13% to 33% depending on scenario, pose gains from 7% to 22%. The ground-truth interactions are defined by an automatic pipeline (Detic + WHAM + Llama), so the benchmark measures agreement with that heuristic stack; that is fine as long as it is acknowledged, and it mostly is.\n\nOverall: a useful new task and dataset, with a central empirical comparison that needs a clarifying pass. I'd encourage you to engage with it, not as an authoritative result but as a starting point for a task definition. A serious referee should ask for the temporal clarification, error bars, and a rerun of baselines with access to the same environment context.\n\nRecommendation: send to peer review, conditional on those issues being addressed.","headline":"New task and benchmark are real, but the central comparison is undermined by an unstated full-take object map, missing error bars, and an overblown abstract.","tokens_in":21867,"tokens_out":3851,"would_cite":true,"duration_ms":35871,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FICTION predicts the 3D locations and body poses of all future object interactions for three minutes ahead, from a 30-second video observation plus a voxel scene map, beating six adapted baselines by more than 30 percent.","keywords":["4D future interaction prediction","egocentric video","3D voxel scene representation","human-object interaction anticipation","SMPL body pose distribution","conditional VAE","long-horizon anticipation","EgoExo4D"],"falsifier":"Train and evaluate FICTION twice, once with the voxel scene map $S$ built only from frames at or before $\\tau_o=30$ s and once with $S$ built from all frames, and count how many object boxes in $S$ come from objects whose first appearance in the video is after 30 seconds. If PR-AUC drops materially when $S$ is cropped to the observed period, the prediction is partly reading the future object inventory from the map rather than forecasting it.","tokens_in":20725,"feed_emoji":"🎯","tokens_out":12379,"duration_ms":99207,"temperature":0.7,"pith_summary":"FICTION predicts the full spatial and postural future of a human activity: which objects in a 3D room the person will touch in the next three minutes, where each interaction will occur, and what distribution of body poses they will use. The paper's central claim is that a 30-second egocentric video observation, combined with an explicit voxel map of object locations and the actor's past body pose, contains enough intent to forecast the rest of the procedure. On three procedural domains from EgoExo4D (cooking, bike repair, health), FICTION reports PR-AUC of 21.0, 18.7, and 12.7 versus best adapted baselines of 16.9, 14.1, and 11.2, and pose error of 229 mm, 372 mm, and 172 mm versus 264 mm, 410 mm, and 221 mm, which it counts as more than 30 percent relative gains. Assistive robots and AR coaching systems would care because such a forecast tells them not just what will happen but where to look, where to stand, and how the person's body will be engaged before the interaction begins.","feed_headline":"Foresight model predicts 3 minutes of future object interactions","feed_subtitle":"FICTION fuses 30 seconds of video with a voxel scene map, beating six baselines by more than 30 percent.","key_machinery":"The load-bearing mechanism is the multimodal transformer encoder that fuses three streams: visual features from the observed video, the actor's SMPL pose parameters, and a $N \\times N \\times N$ voxelized scene map $S$ in which occupied cells carry object-class indices and the actor's location is marked with a reserved index. A linear decoder maps the fused representation onto a future-interaction voxel grid trained with binary cross-entropy, and a conditional VAE takes the same representation plus a query location and produces a Gaussian latent space from which SMPL body poses can be sampled. The voxel grid is what ties 'where' and 'how' together; in the ablations, removing the environment stream causes the largest performance drop.","core_discovery":"The paper's central discovery is that early fusion of three signals, past video, past SMPL body pose, and a persistent 3D voxel representation of object locations, lets a single transformer predict both components of future interaction in a shared 3D space. The location branch decodes the fused representation into a voxel grid marking every 3D cell that will be touched in the next 180 seconds; the pose branch is a conditional VAE that, given a query location, samples the body pose distribution likely to be executed there. On the EgoExo4D procedural benchmark, this beats six adapted baselines: autoregressive models drift and diverge over the three-minute horizon, while video-to-3D scene models without explicit activity grounding predict less accurately. Ablations support the same conclusion: removing the scene map drops PR-AUC from 21.0 to 9.9 on cooking, while removing video or pose degrades it less.","pith_inferences":["Our inference: the paper does not state whether the 3D object map $S$ is built only from frames up to $\\tau_o = 30$ s or from the whole video, so if $S$ contains objects that appear after the observation, some of the reported prediction could be reading the future object inventory from the map.","Our inference: if the method retains its gains when $S$ is strictly cropped to the observed period, the model is probably learning procedural scripts such as 'fridge, then faucet, then cabinet' rather than physical dynamics; swapping object layouts in otherwise identical procedures would separate those two mechanisms.","Our inference: the pose evaluation scores the closest of five sampled poses to the ground truth, so a model that spreads its samples widely is rewarded; a stricter utility measure would use expected error or calibrated diversity over all samples.","Our inference: the most natural deployment setting for this input design is a robot that already knows the room layout from a prior scan, where the object map is genuinely available before the activity starts; the paper's setup implicitly assumes such a map."],"forward_implications":["If FICTION is right, explicit 3D scene conditioning is the ingredient that makes minute-scale interaction anticipation work: removing the object map in the ablations causes a larger drop in PR-AUC than removing either the video or the pose stream.","Autoregressive methods that generate the next token in a sequence are shown to accumulate errors and diverge over the three-minute horizon, which positions direct decoding into a voxel grid as the better architecture for long-horizon forecasts.","The released dataset and fixed task protocol ($\\tau_o = 30$ s observation, $\\tau_a = 5$ s anticipation gap, $\\tau_f = 180$ s future, with PR-AUC, Chamfer distance, MPJPE, and PA-MPJPE) give subsequent work a public benchmark for 4D future interaction prediction.","Assistive agents could turn the forecast into concrete preparation: knowing the interaction location and likely pose distribution, a robot or AR coach can position itself and cue the right assistance before the contact happens."],"supporting_citations":[{"why":"provides the EgoExo4D raw videos and take-level train/test split from which the 4D interaction dataset is built.","marker":"[38]"},{"why":"serves as the long-term anticipation autoregressive baseline that FICTION must beat.","marker":"[5]"},{"why":"is the interaction-hotspot baseline whose 2D predictions are lifted to 3D for comparison.","marker":"[63]"},{"why":"is the autoregressive human-pose baseline used in the pose prediction comparison.","marker":"[35]"},{"why":"is the text-conditioned pose generation baseline, including its fine-tuned version T2M-GPT-FT.","marker":"[111]"},{"why":"is the sparse voxel transformer baseline adapted from 3D scene completion to future interaction prediction.","marker":"[55]"},{"why":"is the video-to-3D occupancy transformer baseline adapted to this task.","marker":"[114]"},{"why":"supplies the conditional VAE architecture that generates the pose distribution at a query location.","marker":"[93]"},{"why":"Detic segments objects in video frames, which are then mapped to 3D to build the voxel scene map.","marker":"[118]"},{"why":"WHAM extracts the SMPL body poses that serve as observed input and ground-truth pose targets.","marker":"[92]"}],"fun_headline_variants":["4D foresight: predicting where and how people interact next","3-minute future: model predicts 3D locations and poses of interactions","FIction: video to 4D interaction prediction, beating 2D baselines by 30%","Anticipating interactions in 3D: early fusion of video, pose, and scene map"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never states whether the 3D object map is built only from video frames up to the $\\tau_o=30$ second observation time or from the whole video, so objects that appear only after the observation could be leaking future scene content into the model's input.","fun_headline_variants_meta":{"raw":{"variants":["4D foresight: predicting where and how people interact next","3-minute future: model predicts 3D locations and poses of interactions","FIction: video to 4D interaction prediction, beating 2D baselines by 30%","Anticipating interactions in 3D: early fusion of video, pose, and scene map"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2924,"prompt_tokens":923,"completion_tokens":2001,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1912}},"tokens_in":539,"tokens_out":2001,"duration_ms":14539,"temperature":1.0,"reasoning_tokens":1912,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:51:00.589588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train and evaluate FICTION twice, once with the voxel scene map $S$ built only from frames at or before $\\tau_o=30$ s and once with $S$ built from all frames, and count how many object boxes in $S$ come from objects whose first appearance in the video is after 30 seconds. If PR-AUC drops materially when $S$ is cropped to the observed period, the prediction is partly reading the future object inventory from the map rather than forecasting it.","supporting_citations":[{"cited_title":"Joint hand motion and interaction hotspots prediction from egocentric videos","cited_arxiv_id":null,"evidence_quote":"is the interaction-hotspot baseline whose 2D predictions are lifted to 3D for comparison."},{"cited_title":"Generating human motion from textual descrip- tions with discrete representations","cited_arxiv_id":null,"evidence_quote":"is the text-conditioned pose generation baseline, including its fine-tuned version T2M-GPT-FT."},{"cited_title":"Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar","cited_arxiv_id":null,"evidence_quote":"is the sparse voxel transformer baseline adapted from 3D scene completion to future interaction prediction."},{"cited_title":"Occformer: Dual-path transformer for vision-based 3d semantic occu- pancy prediction","cited_arxiv_id":null,"evidence_quote":"is the video-to-3D occupancy transformer baseline adapted to this task."},{"cited_title":"Learn- ing structured output representation using deep conditional generative models","cited_arxiv_id":null,"evidence_quote":"supplies the conditional VAE architecture that generates the pose distribution at a query location."},{"cited_title":"Detecting twenty-thousand classes using image-level supervision","cited_arxiv_id":null,"evidence_quote":"Detic segments objects in video frames, which are then mapped to 3D to build the voxel scene map."},{"cited_title":"Wham: Reconstructing world-grounded humans with accurate 3d motion","cited_arxiv_id":null,"evidence_quote":"WHAM extracts the SMPL body poses that serve as observed input and ground-truth pose targets."}],"review_version":1}