{"id":"c71e406d-5570-492d-86b4-42183026cc39","arxiv_id":"2608.01049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FactorJEPA splits a video prediction model into layout, agent, and interaction channels with a visibility gate, and a new DENSEWORLD dataset tests it on crowded Indian city scenes.","lead":"This paper introduces DENSEWORLD, a 1,000 hour video dataset of crowded Indian cities, and FactorJEPA, a method that predicts future video by separating layout, agents, and their interactions instead of one monolithic prediction channel. The authors report better future prediction and intervention sensitivity for V-JEPA world models when trained on this data, especially at the full 115k clip scale.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Audit-split contradiction threatens the independent evaluation behind intervention-sensitive prediction; the paper must clarify whether a human-annotated audit split actually exists.","rationale":"The Reader's weakest assumption is that DINOv2-derived factor targets are reliable enough to supervise the predictor. My stress-test centers on the related but sharper issue: the paper's own text contradicts whether an independent human-annotated audit split exists. This is load-bearing because Causal L1, a headline metric, depends on independent intervention regions from that audit. If the audit split does not exist or is not actually human-annotated, the central claim loses its evaluator-independence guarantee for one of its three pillars. If the audit split does exist, the concern does not land, and the paper still has strong internal controls (FactorJEPA-RAW, frozen target encoders, paired clustered inference). The reader already issued CONDITIONAL, so my concern reinforces that verdict rather than moving it; hence UNCHANGED. I flag this as a concrete, verifiable issue rather than a general suspicion: the discrepancy between Ψ_human in the main text and \"no manually annotated factor labels\" in the appendix is directly observable and should be settled by releasing and inspecting the audit annotations.","tokens_in":45524,"tokens_out":5930,"duration_ms":64818,"concrete_test":"Download the released DENSEWORLD-115k artifacts and verify the audit split: count clips containing Ψ_human annotations, check that D_audit is city-disjoint from D_train, and reproduce Figure 8's Causal L1 values using those annotations. Then rerun Causal L1 with intervention regions generated by the DINOv2 pipeline instead of the audit annotations. If the audit files are absent, or if FactorJEPA's Causal L1 advantage over Auto-RGN falls below the 1× CI separation threshold under DINOv2-generated interventions, the intervention-sensitivity claim is not independently established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, the headline diagnostics must be evaluated against signals independent of the DINOv2 teacher. The main text's \"Evaluator Independence and Audit Split\" section defines a city-disjoint audit split D_audit whose agent masks, visibility states, interaction pairs, and intervention regions are \"independently annotated\" via T_audit = Ψ_human(x). But Appendix A.3 states that \"DENSEWORLD does not use manually annotated factor labels\" and that all targets are generated by the fixed DINOv2 pipeline; Appendix C repeats \"no manually annotated factor labels.\" These statements cannot both be true. If the Ψ_human audit split does not exist, Causal L1—one of the three headline improvements—has no independent intervention regions, and the paper's safeguard against \"agreement with the pseudo-label generator being mistaken for improved world modeling\" collapses. The paper's own limitation (i) concedes that interaction grounding is not fully solved, so the audit is the only stated mechanism for bounding that bias. The contradiction must be resolved before the intervention-sensitivity claim is clean.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DENSEWORLD, a roughly 1,000-hour, 22-city video benchmark for crowded, chaotic Global South urban scenes, and FactorJEPA, a variant of V-JEPA in which the monolithic future predictor is replaced by factorized layout, agent, and interaction channels with a soft visibility gate and separation losses. The central empirical claims are that FactorJEPA improves future-latent accuracy (Future-frame L1), intervention-sensitive prediction (Causal L1), and robustness to reduced evidence (Mask-ratio slope) over conventional fine-tuning and parameter-efficient adaptation baselines, with stable method rankings across 2B and 1B V-JEPA 2.1 backbones. The paper includes a detailed appendix with target-construction contracts, training pseudocode, resource accounting, paired clustered inference, ablations, and a latent-to-RGB decoding analysis, and it releases the dataset and checkpoints.","tokens_in":45873,"tokens_out":7795,"duration_ms":90430,"significance":"If the empirical claims hold, the paper makes a useful contribution: it identifies a predictive regime that is underrepresented in JEPA evaluations, provides a large public dataset for that regime, and shows that explicitly structured predictive channels can outperform monolithic adaptation for dense, partially observed urban scenes. The paper's strengths include a carefully matched experimental protocol across methods, paired hierarchical bootstrap inference, a clear attribution chain (Auto-RGN -> FactorJEPA-RAW -> FactorJEPA), unusually explicit limitation statements, and public release of data and checkpoints. The main weaknesses are not in the experimental ambition but in unresolved inconsistencies in the definition and provenance of the primary measurement signals, which currently prevent the central claims from being accepted as stated.","major_comments":[{"comment":"The main text states that the audit split D_audit has agent masks, visibility states, interaction pairs, and intervention regions 'independently annotated' via T_audit = Ψ_human(x), and Limitations (i) relies on this audit split to ensure the interaction generator is not reused as the headline evaluator. However, Appendix A.3 states that 'DENSEWORLD does not use manually annotated factor labels' and Appendix C.1 repeats 'no manually annotated factor labels.' These statements are mutually contradictory. If Ψ_human does not exist, Causal L1 has no human-annotated intervention regions and the stated safeguard against agreement with the DINOv2-based pseudo-label generator collapses. The paper must state clearly whether a human-annotated audit split exists; if it does not, it must specify the actual provenance of the intervention regions (e.g., the automatic G_int in Appendix E.2) and demonst","section":"§4 'Evaluator Independence and Audit Split'; Appendix A.3; Appendix C.1; Limitations (i)"},{"comment":"The main text and figures (including Figure 8 and Figure 9) report the headline metric as 'Future-frame L1' and define it as a normalized L1 distance. Appendix E.2, however, defines 'Future-frame MSE' as the executed quantity and explicitly says: 'The executed scorecard reports this MSE quantity; it is not relabeled as L1.' Table 8 also lists 'Future-frame MSE' as the primary metric. Since this metric carries one of the paper's three central claims and the cross-scale correlations, the discrepancy is load-bearing. The authors must specify which quantity is actually computed and reported, and correct the main text, appendix, or tables accordingly. If MSE is used, all reported values and comparisons need to be re-expressed or relabeled consistently.","section":"§4 'Future-frame L1' vs Appendix E.2 and Table 8"},{"comment":"The main text says Future-frame L1 and Mask-ratio slope use 'the frozen V-JEPA target encoder,' which suggests a single, method-independent evaluation target. But the training protocol (Algorithm .1 step 41 and Table 6) updates the momentum target encoder by EMA from the online encoder, so after training the target encoder is generally not the original pretrained V-JEPA checkpoint unless a separate frozen copy is used for evaluation. If evaluation uses the EMA-updated target encoder, then each method is evaluated against a different target space and the claimed independence of the headline diagnostics is not established. The paper must specify exactly which target encoder is used at evaluation time: a frozen pretrained V-JEPA encoder, or the per-method momentum encoder. This matters for Future-frame L1, Mask-ratio slope, and Causal L1 alike.","section":"§4 'Future-frame L1' and 'Mask-ratio slope'; Protocol, Algorithm .1 step 41; Table 6"}],"minor_comments":[{"comment":"The sentence 'As shown in Section , we measure this regime...' has an empty section reference. Please fill in the correct section number.","section":"§2 'How Dense is DENSEWORLD?'"},{"comment":"The text says 'Figures 17 and 17 show all executed diagnostics for the ViT-G and ViT-g backbones.' The figure numbering appears duplicated/broken; please renumber and reference the scorecards uniquely.","section":"Appendix E.5"},{"comment":"Several resource entries are labeled 'prespecified execution ceilings' and 'must not be described as measured consumption unless verified against profiling logs.' As currently presented, the table mixes commitments with measurements. Please separate budget ceilings from profiled results, or state explicitly in the main text that resource figures are planned envelopes, not measured outcomes.","section":"Table 6"},{"comment":"The metric is called 'Causal L1' in the abstract and main text but 'Intervention L1' in Table 8, where the paper notes the revised name reflects that the metric does not establish causal identification. Align the terminology throughout to avoid confusion.","section":"§4 and Table 8"}],"recommendation":"major_revision","confidential_remarks":"The three major comments are resolvable in principle, but they touch the exact quantities on which the central empirical claims rest. I would ask the editor to require a point-by-point response on the audit-split existence, the L1/MSE identity, and the evaluation target-encoder protocol before considering the paper again. The paper's experimental infrastructure and transparency are otherwise strong; this is not a case of methodological incompetence, but of under-specified measurement provenance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. The first is that the core contribution is real: a factorized JEPA predictor with a soft visibility gate, separated subspaces, and a staged curriculum, plus a 1,000-hour Indian urban video dataset. The evaluation is unusually thorough – matched baselines, ablations, cross-scale checks, a clean attribution chain from frozen V-JEPA to FactorJEPA-RAW to full FactorJEPA, and they release the dataset and checkpoints. The limitations section is unusually honest. That's rare enough to be worth a serious look.\n\nThe second thing is that the headline claim about intervention-sensitive prediction is currently undermined by an internal contradiction. In the main text's \"Evaluator Independence and Audit Split\" section, they define a city-disjoint audit split with agent masks, visibility states, interaction pairs, and intervention regions \"independently annotated\" as T_audit = Ψ_human(x). Appendix A.3 states that DENSEWORLD does not use manually annotated factor labels, and Appendix C repeats \"no manually annotated factor labels.\" Those two statements cannot both be true. The appendix is explicit; the main text is vague. If the human-annotated audit split does not exist, then Causal L1, one of the four headline diagnostics, no longer has independent intervention regions, and the paper's stated safeguard against \"agreement with the pseudo-label generator being mistaken for improved world modeling\" collapses. The stress-test note is on target. This has to be fixed: either show the human annotations or re-run Causal L1 with an independent intervention generator.\n\nMinor issues, not fatal: the primary metric is called Future-frame L1 in the abstract and main text but Future-frame MSE in Appendix E; and the teacher pipeline's reliability is plausible but not externally validated. The paper's own limitation (i) admits interaction grounding is not fully solved, which is fine, but it makes the audit split the only mechanism bounding that bias.\n\nWho this is for: world-modeling researchers, particularly JEPA and video prediction. The dataset alone is valuable. The factorized predictor will likely be reused. The paper deserves a serious referee, but the referee should have the audit-split contradiction addressed first. I'd send it out, with a note to the authors asking them to clarify or produce the audit data.","headline":"Real contributions in the factorized predictor and DENSEWORLD dataset, but the audit-split contradiction must be resolved before Causal L1 is credible.","tokens_in":46305,"tokens_out":2633,"would_cite":true,"duration_ms":24926,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing a monolithic JEPA predictor with factorized layout, agent, and interaction channels improves future-latent accuracy, intervention sensitivity, and occlusion resilience in dense urban video.","keywords":["world models","JEPA","factorized prediction","dense urban scenes","partial observability","DENSEWORLD","video forecasting","intervention sensitivity"],"falsifier":"Train FactorJEPA with the same architecture but feed it temporally shuffled or clip-scrambled layout, agent, and interaction targets. If the predictive advantages over FactorJEPA-RAW survive or grow, the claimed semantic anchoring is not the active mechanism and the gains come from regularization rather than structure.","tokens_in":45494,"feed_emoji":"🚦","tokens_out":5534,"duration_ms":55770,"temperature":0.7,"pith_summary":"The paper argues that JEPA (Joint Embedding Predictive Architecture) world models fail in crowded, chaotic cities because their monolithic predictor packs everything into one entangled latent, letting the model exploit visual shortcuts instead of recovering scene structure. To fix this, FactorJEPA decomposes the predicted future into layout, visibility-gated agents, and sparse interactions, each with its own subspace and supervision. On DENSEWORLD, a new 1,000-hour, 22-city dataset of dense Global South urban video, this factorized predictor lowers future-frame L1 and causal intervention L1, flattens the mask-ratio slope, and leaves a reproducible motion trade-off that full-scale training resolves. If the claim holds, explicit factorization, not just more fine-tuning, is the way to make video world models forecast dense, partially observed scenes.","feed_headline":"FactorJEPA beats monolithic JEPA on dense urban futures","feed_subtitle":"Splitting the future latent into layout, agents, and interactions improves prediction, intervention sensitivity, and resilience under occlus","key_machinery":"The load-bearing object is the factorized predictor with block-structured matrix factorization $\\hat{Y}= C A^\\top = C_L A_L^\\top + C_A A_A^\\top + C_I A_I^\\top$. Each factor has its own coordinate block, synthesis dictionary, and reliability-weighted supervision; a soft visibility gate $v_n^{(i)}$ attenuates uncertain or occluded agents without removing them, and a separation loss penalizes linear and nonlinear cross-channel covariance. This forces the factorization to be architectural rather than a post hoc label: distinct pathways and dictionaries compose the future embedding in a way a monolithic predictor does not.","core_discovery":"FactorJEPA's central claim is that how a predictor organizes future information matters as much as how well it matches the target. In a conventional JEPA, the future embedding is produced by one monolithic predictor, and in dense scenes that predictor can satisfy the objective by mixing crowd texture, visible appearance, and road geometry into shortcuts. FactorJEPA composes the future latent as $\\hat{Y}=C_L A_L^\\top + C_A A_A^\\top + C_I A_I^\\top$, with layout, visibility-gated agent, and sparse interaction coordinates, each anchored by DINOv2-derived targets with reliability weights, and penalizes cross-channel leakage. The paper reports separation from the best matched baseline on Future-fr","pith_inferences":["The factorization principle should transfer beyond Global South streets to other crowded partial-observability settings—indoor crowds, disaster scenes, mixed mobility—provided a teacher can supply layout, agent, and interaction targets.","The separation loss may serve as a practical interpretability diagnostic: if off-diagonal leakage is low, individual channels can be inspected or edited with predictable effects on the decoded future.","Causal L1 only measures consistency with imposed interventions, not causal identification; the strongest reading of the paper is structure-preserving prediction rather than learned causal mechanisms.","A teacher-free variant that discovers layout, agent, and interaction coordinates without DINOv2 targets would test whether the architecture alone, rather than the specific teacher, is the source of the gains."],"forward_implications":["If the factorization is what drives the gains, JEPA world models for dense scenes should be built with explicit layout, agent, and interaction channels rather than monolithic predictors.","The mask-ratio result implies that factorized structure can soften degradation under partial observability, so occluded agents continue to be predicted through visibility gating and interaction priors.","Cross-scale stability means a 1B backbone can stand in for a 2B model when screening adaptation choices, reducing experiment cost.","The motion-cosine trade-off, if data-dependent, implies that factorized training prioritizes motion that supports forecasting, and that the trade-off is not an intrinsic cost of factorization.","DENSEWORLD offers a benchmark stress test for any world model claiming to handle dense urban dynamics and partial observability."],"supporting_citations":[{"why":"Supplies the frozen DINOv2 teacher pipeline that generates layout, agent, visibility, and interaction targets used to anchor FactorJEPA channels.","marker":"Oquab et al. 2024"},{"why":"Provides the LoRA baseline, a matched parameter-efficient adaptation method that FactorJEPA must beat.","marker":"Hu et al. 2022"},{"why":"Provides the DoRA baseline, a weight-decomposed low-rank adaptation method compared against.","marker":"Liu et al. 2024"},{"why":"Provides the Auto-RGN gradient-norm surgical fine-tuning method, the primary matched monolithic reference.","marker":"Lee et al. 2023"},{"why":"Provides the BDD100K benchmark used for matched density and occupancy comparisons that establish DENSEWORLD as a distinct regime.","marker":"Yu et al. 2020"},{"why":"Provides the nuScenes benchmark used for matched density and occupancy comparisons that establish DENSEWORLD as a distinct regime.","marker":"Caesar et al. 2020"}],"fun_headline_variants":["FactorJEPA splits future into layout, agents, interactions","Factorized futures beat monolithic JEPA in dense urban scenes","Decomposing future latent improves prediction in chaotic cities","FactorJEPA: layout, agents, interactions for crowded urban worlds","Predicting bustling cities: FactorJEPA factors the future"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole gain depends on the DINOv2-derived teacher targets being reliable enough to anchor the three channels; if they miss small or occluded agents or misclassify interactions, the factor supervision can bias the predictor rather than improve it.","fun_headline_variants_meta":{"raw":{"variants":["FactorJEPA splits future into layout, agents, interactions","Factorized futures beat monolithic JEPA in dense urban scenes","Decomposing future latent improves prediction in chaotic cities","FactorJEPA: layout, agents, interactions for crowded urban worlds","Predicting bustling cities: FactorJEPA factors the future"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1370,"prompt_tokens":927,"completion_tokens":443,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":671,"tokens_out":443,"duration_ms":5277,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:34:42.507713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train FactorJEPA with the same architecture but feed it temporally shuffled or clip-scrambled layout, agent, and interaction targets. If the predictive advantages over FactorJEPA-RAW survive or grow, the claimed semantic anchoring is not the active mechanism and the gains come from regularization rather than structure.","supporting_citations":[],"review_version":1}