{"id":"d599feaa-1ef4-4c16-948d-3eca945143e1","arxiv_id":"2608.09449","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A 128,892-clip, 2,826-hour real-world video dataset with camera trajectories, temporally grounded captions, and 982 revisit-rich panoramic sequences for interactive world-model training.","lead":"Sekai2 is a new video dataset with nearly 129,000 real-world clips, about 2,800 hours total, each paired with a camera trajectory and time-stamped scene descriptions. It adds 982 panoramic walks with loops and revisits, which could help train computer models that remember places across viewpoints and time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline claim overstates the release: the 982-panorama/119-hour revisit subset is mostly not publicly available (Sec. 6), so the field cannot currently use the central supervision.","rationale":"The reader's verdict is already CONDITIONAL, and I would keep it conditional, so no verdict change is needed. My partial agreement reflects a different emphasis: the reader's weakest assumption is the loop-closure verification in App. B.4, while I see the limited public release of the panoramic subset as the more load-bearing issue. The strongest claim is about what the field gains, and most of the 982/119-hour revisit set is not downloadable; Sec. 6 explicitly says only about 20 hours can be released. The loop-closure concern is technically plausible: ORB matching plus a rotation-only median residual over spherical rays can confuse near-coincident but distinct camera centers in visually repetitive environments such as malls and corridors, and no precision/recall or metric ground truth is reported for closure acceptance. But that concern would only be testable and impactful if the data were available. The release gap is objective, disclosed, and directly contradicts the abstract and Table 1. This is a condition on acceptance: either release the full panoramic set under a compliant license, or revise the headline and corpus statistics to clearly separate the processed internal corpus from the public release. The paper's other evidence, including epipolar validation, cross-run DROID consistency with matched intrinsics, and detailed manifest provenance, does not override this condition.","tokens_in":33663,"tokens_out":11668,"duration_ms":115764,"concrete_test":"Download the public release manifest and media/pose/caption files from the project page and count panoramic clips and total hours; compare with Table 1 and Sec. 4.2. If the public manifest contains fewer than 982 panoramic sequences or about 119 hours, the 'release contains' claim is not satisfied and the abstract/Table 1 must be revised to distinguish the processed corpus from the public release. If all 982 sequences and 119 hours are actually downloadable, this concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract and Table 1 state that the 'release' contains 128,892 clips totaling 2,826 hours, including 982 panoramic sequences totaling 119 hours with loops and revisits. Section 6, 'Panoramic data release,' states that the current public release can include only approximately 20 hours of this material due to licensing constraints, with all reported panoramic statistics and analyses based on the full 119 hours. The publicly downloadable corpus is therefore about 2,727 hours, not 2,826 hours, and the distinctive revisit-rich panoramic supervision is mostly inaccessible: users cannot obtain most of the 982 sequences, their camera trajectories, or their captions. The paper is transparent about this limitation, but the headline is internally inconsistent with the released artifact, and the most novel component cannot be independently verified or used. The reader's loop-closure concern (rotation-only verification over spherical rays in repetitive scenes) is a real secondary risk, but it is subordinate to the availability gap: even perfect loop closures would not make unavailable data usable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Sekai2, a multi-source real-world video dataset aimed at interactive world modeling. It combines inherited Sekai footage, newly crawled YouTube videos, and self-captured panoramic videos; applies a manifest-driven filtering and quality-control pipeline; estimates camera trajectories with ViPE and DROID-SLAM; generates hierarchical global and segment-level captions with a VLM; and reports corpus-scale evaluations of visual quality, trajectory geometry, caption quality, and cross-modal grounding. The headline contribution is the joint provision of long-horizon continuity, camera-pose supervision, temporally grounded semantics, and a revisit-rich panoramic subset with loop-closure-refined trajectories.","tokens_in":33783,"tokens_out":5998,"duration_ms":55838,"significance":"If fully released as described, Sekai2 would be a valuable resource for long-horizon video generation, camera-controllable synthesis, and world-model pretraining. The paper has real strengths: the manifest-driven pipeline is documented in unusual detail with stage-wise retention statistics; the epipolar validation includes a mismatched-pose negative control that genuinely isolates pose-frame correspondence; the cross-run DROID consistency study is carefully matched in intrinsics and motion regime; and the cross-modal grounding analysis uses within-clip controls. These make the perspective-video portion of the corpus credible and reproducible. At the same time, the paper's most distinctive component—the 982-sequence, 119-hour panoramic revisit subset—is mostly not publicly available, and its trajectories receive only qualitative geometric validation. These gaps substantially temper the significance as claimed.","major_comments":[{"comment":"The abstract and Table 1 state that the release contains 982 panoramic sequences totaling 119 hours with loops and revisits, and §4.2 reports the panoramic subset as 119 hours. However, the 'Panoramic data release' paragraph in §6 states that only approximately 20 hours of this material can be included in the current public release, with all reported panoramic statistics based on the full 119 hours. The publicly downloadable corpus is therefore about 2,727 hours, not 2,826 hours, and most of the distinctive revisit-rich trajectories, their poses, and their captions are not accessible to users. This overstates the released artifact and prevents independent verification of the paper's most novel component. The paper should distinguish the processed internal corpus from the publicly downloadable release, and should restate the headline numbers, Table 1 entries, and all panoramic statistics for the actually released subset.","section":"§6 (Panoramic data release) vs Abstract/Table 1/§4.2"},{"comment":"The quantitative geometric validation of trajectories excludes the panoramic subset. Table 3 states that panoramic sequences are excluded because the reconstruction assumes pinhole imagery, and the epipolar test in §5.2 samples only the perspective subsets; for the panoramic sequences the only quantitative evidence is numerical validity/smoothness (Table 2) and internal loop statistics, supplemented by qualitative full-accumulation figures (App E.4). This is a load-bearing gap because the loop-closure acceptance criterion in App B.4 is a low median angular residual under a rotation-only model, which can also be satisfied by near-coincident viewpoints or by repeated similar geometry in malls and corridors. The paper should add a quantitative validation of the loop-refined panoramic trajectories, for example against manually verified revisit events or a synthetic baseline, and should report the false-positive behavior of the rotation-only verification.","section":"Table 3 / §5.2 / App B.4"},{"comment":"The caption-quality evaluation is partly self-referential. In the pairwise comparison, the structured annotations and the one-shot comparator are generated by the same VLM from identical frames, and the judge model is not named; if the judge is the same model family used to produce the annotations, the 60.9% preference and 68.4% informativeness results may reflect the judge's familiarity with its own output style rather than an independent quality advantage. Holding the visual input fixed does not remove this potential bias. The authors should disclose the judge model and report either a cross-model judge or a human-sample evaluation, and should present the current results as an internal consistency check.","section":"§5.3 / Table 4"}],"minor_comments":[{"comment":"The text contains stray commas in '0.12,px' and '0.5,s'; these should read '0.12 px' and '0.5 s'.","section":"§5.2"},{"comment":"The annotation model is called 'Kimi-K2.6' in the text, but reference [39] is titled 'Kimi k2.5: Visual agentic intelligence'; the version designation should be aligned.","section":"§3.5 / Ref. [39]"},{"comment":"Figure 1's caption gives 1,912 hours for the newly collected YouTube subset while the figure's internal label and §4.2 give 1,911 hours; these numbers should be reconciled.","section":"Fig. 1 / §4.2"},{"comment":"The phrase 'a evaluation using a fixed prior focal length' should read 'an evaluation using a fixed prior focal length'.","section":"§5.2"},{"comment":"In the rows for WebVid, Panda-70M, and OpenVid-1M, the 'Pose' column appears to use a combined '–✓' marker; separate ✓/✗ entries or a clear legend would avoid ambiguity.","section":"Table 1"},{"comment":"The statement that the procedure 'handles mid-trajectory revisits rather than only start-to-end returns, including 25 such sequences' would benefit from explaining how the 25 mid-trajectory revisits were identified, since the candidate-generation description emphasizes keyframes from the beginning and ending portions.","section":"App B.4"}],"recommendation":"major_revision","confidential_remarks":"The availability gap is the decisive issue. The authors are transparent in §6, but the abstract and Table 1 still claim a release of 2,826 hours including 119 hours of panoramic revisits, while only about 20 hours of panoramic data are publicly downloadable. If dataset availability is treated as a core criterion for this venue, this is close to a reject; I recommend major revision because the framing can be corrected and the public subset can be characterized separately. The loop-closure verification concern is secondary but should be addressed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Sekai2 is a serious dataset effort and the most honest one I have read in this space, but you need to know that the centerpiece — the 982-panorama revisit-rich subset — is not actually in the public release. The abstract and Table 1 advertise 2,826 hours including 119 hours of panoramas; Section 6 says only ~20 hours of that material can be distributed under current licensing. So the actual download is roughly 2,727 hours, and the loop/revisit supervision that motivates the whole paper is mostly invisible. Credit where it is due: the paper is transparent about this in the limitations section, and the rest of the pipeline is described in unusual detail — shot-aware clipping, filtering gates, pose QC, annotation schema, all with thresholds and appendix-level recipes. The epipolar validation with a mismatched-pose negative control is a genuine check: 0.43 px median error versus 2.83 px when the pose pair is shifted, and the motion-stratified control analysis is thoughtful. That is credible image-based evidence that poses and frames are aligned. Cross-run DROID consistency is a reasonable auxiliary check, and they flag the ViPE-same-stack caveat themselves. The corpus statistics are internally coherent and the attribute-dependence analysis is a nice touch beyond the usual histograms.\n\nThe soft spots are real but not fatal to the data-engineering contribution. First, the availability gap I started with: all panoramic statistics and the revisit-based supervision claims rest on the full 119 hours, which you cannot download. That makes the most novel claim unverifiable right now. Second, the caption-quality evaluation is partly self-referential: Kimi generates captions, Kimi judges them, and the pairwise comparison uses the same VLM as one-shot comparator and judge. The position-bias reporting is honest, and the vocabulary complementarity is at least a structural sanity check, but the 'quality' claim needs human or independent instrumented evaluation. Third, the loop-closure refinement uses rotation-only alignment over spherical rays; in corridors or malls with repeated geometry, a low angular residual can come from near-coincident but distinct viewpoints. The paper's geometric verification with RANSAC-like consistency helps, but does not settle the co-location question. Since the panoramic data is mostly unreleased, this is currently a risk note rather than a demonstrated flaw. Also, no world model is trained, so 'scalable resource for interactive pre-training' is asserted, not demonstrated. The paper says this is inference-only evaluation, and the field can accept that for a dataset, but the claim is not load-bearing in the data-engineering sense.\n\nWho this is for: researchers building long-horizon camera-conditioned generators and people designing data manifests for world models. It deserves a serious referee because the pipeline description and evaluation design are ahead of the usual dataset paper, and the transparency about failures is exemplary. My recommendation: send it to peer review, but require that the abstract and released-artifact description be corrected to state the actual public size, and ask for a human-annotated caption sample or an independent judge before acceptance.","headline":"A serious, unusually well-documented dataset paper whose headline claim is undercut by the public release: the 982-panorama revisit-rich subset is mostly not downloadable, so the central supervision is currently out of reach.","tokens_in":34438,"tokens_out":1832,"would_cite":false,"duration_ms":17713,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sekai2 supplies world models with what real-world corpora have lacked: 2,826 hours of video where every clip has a camera trajectory, timed captions, and loops that revisit the same places.","keywords":["video dataset","interactive world models","camera trajectories","temporally grounded captions","long-horizon video generation","panoramic video","loop closures","revisit consistency"],"falsifier":"Take a random sample of accepted loop-closure pairs and fit a full essential matrix (rotation plus translation) to their matched spherical rays instead of a rotation-only alignment. A genuine co-location from the same camera center should show negligible translation; if a substantial fraction of accepted closures show significant parallax—estimated camera-center separation above a few percent of scene depth—the revisit labels are partly spurious and the pose-graph constraints would bias the trajectories. A second ground-truth check: for sequences designed as start-to-end loops, compare the refined start and end camera centers; if endpoint displacement does not drop after loop-closure refinement relative to the original ViPE estimate, the verification is not doing the claimed work.","tokens_in":33347,"feed_emoji":"🎥","tokens_out":14660,"duration_ms":123253,"temperature":0.7,"pith_summary":"This paper argues that training interactive world models—generative systems that keep a scene's state while the camera moves—requires video data offering long-horizon continuity, camera poses, and time-localized semantics all at once, and that existing corpora supply these only in fragments. To close that gap, it builds and releases Sekai2: 128,892 clips totaling 2,826 hours from 10,428 source videos across 113 countries, with every clip paired with a camera trajectory and hierarchical annotations yielding 649,597 temporally grounded segments. The added element is a 982-sequence panoramic subset captured along non-linear routes with loops and revisits, providing repeated observations of the same locations across time and viewpoint—supervision the authors argue is needed for persistent scene representations and long-term spatial memory. A sympathetic reader would take the central claim to be that the joint provision of these four properties in one real-world corpus, rather than any single dimension of scale, is what makes Sekai2 a foundation for long-horizon video generation, camera-controllable synthesis, and world-model pre-training.","feed_headline":"2,826 hours of video join long clips, camera paths, and revisit loops","feed_subtitle":"2,826 hours of real footage with timed captions give world models a training ground for long-horizon consistency.","key_machinery":"The load-bearing mechanism is the unified multi-source data engine, but the object that carries the paper's most distinctive claim is the revisit-verification procedure for panoramic trajectories. A revisit candidate is proposed by ORB feature matching between temporally distant keyframes; the matched features are lifted to unit viewing rays on the sphere, a rotation-only alignment is fit by singular value decomposition, and the candidate is accepted only when the median angular residual is low—the logic being that a genuine co-location is explained by a single rotation, whereas a changed camera center produces parallax no rotation can remove. Accepted closures enter a pose-graph optimization as camera-center constraints with orientation left free, correcting accumulated drift over roughly 13,000-frame sequences. The complementary machinery is the hierarchical annotation schema: a single vision-language-model call produces clip-level and segment-level JSON with six shared text fields, a sixteen-value camera_path vocabulary, and full/short prompt pairs, so camera control and scene semantics can be separated at training time. A cross-modal audit—comparing pose-derived yaw change with the caption's camera_path label over 620,487 aligned segments—is the paper's end-to-end check that the two independently produced modalities are grounded in the same events.","core_discovery":"On the paper's own terms, the discovery is that a real-world video corpus can be engineered so that visual content, camera motion, and localized semantic change are aligned on a shared timeline at corpus scale. Every one of the 128,892 released clips is accompanied by a ViPE-estimated camera-to-world trajectory and intrinsics, and by two levels of vision-language-model captions: clip-level global attributes and segment-level local descriptions that tile each video and carry a discrete camera_path label. The annotation schema factorizes each description into subject motion, environment motion, static scene content, and camera behavior, with camera clauses enclosed in camera tags so they can be dropped, replaced, or conditioned on independently. The paper further claims that the 982 panoramic sequences, refined by geometrically verified loop closures embedded in a pose graph, deliver genuine revisits—repeated observations of the same locations from different directions and at different times—that prior real-world corpora lack. Corpus-wide evaluations are offered as evidence: all released trajectories pass validity and smoothness checks, epipolar error against SIFT correspondences is sub-pixel with a mismatched-pose control degrading sharply, cross-run DROID reconstructions agree to roughly 0.1 degrees of rotation at one second, caption scores sit near 4.3 on a 1–5 judge scale, and pose-derived yaw increases monotonically across the stationary–straight–mixed–turn camera labels.","pith_inferences":["A benchmark opportunity the paper does not pursue: the monotonic rise in measured yaw across the caption labels (stationary, straight, mixed, turn) could be inverted into a ground-truth-free consistency score for generated videos, rewarding models whose implied camera motion agrees with their own captions.","The factorization into subject motion, environment motion, static scene, and camera behavior sets up a disentanglement experiment the paper does not run: hold scene content fixed while varying only the camera-conditioning input, and test whether a model separates viewpoint change from world change.","Because data-distribution agreements limit the public panoramic release to roughly 20 of the 119 processed hours, the corpus's most distinctive supervision is only partially downloadable; the reported statistics describe the full processed set, so downstream users should verify which portion they actually receive.","If the revisit supervision proves dense enough, it could shift long-range memory research away from game and simulated environments toward real-world trajectories, since Sekai2 couples repeated observations with captured surroundings—but whether that supervision transfers into training gains remains untested by this paper."],"forward_implications":["Camera-conditioned video generation can now train on real-world trajectories: every released clip ships poses and intrinsics convertible to ray or Plücker embeddings, so camera control need not be learned only from synthetic or short-range reconstruction sources.","Long-horizon training data is no longer confined to short clips: 43,594 analysis segments reach the full 120-second cap and 92.3% of footage lies in segments of at least one minute, giving world models sustained observation for learning scene persistence.","Revisit supervision becomes available for real-world scenes: the 982 panoramic sequences with loops and repeated passages offer repeated observations of the same locations, the scarce signal identified by prior work as needed for long-range spatial memory and loop-closure behavior.","Semantic and geometric supervision can be separated or combined at will: the camera-tagged full prompts and the camera-free short prompts support three training modes from one caption corpus, and discrete camera_path labels align caption segments with measured yaw changes at an AUC of 0.709."],"supporting_citations":[{"why":"Supplies the inherited 47,699 clips and the world-exploration lineage that Sekai2 reprocesses; establishes the predecessor corpus the new release extends.","marker":"[28]"},{"why":"ViPE is the pose engine whose pose-only deployment produces every released camera trajectory and intrinsics, so the entire geometric-supervision claim depends on it.","marker":"[19]"},{"why":"DROID-SLAM performs the SLAM optimization inside the pose pipeline and is the reconstruction engine for the cross-run geometric-consistency evaluation.","marker":"[42]"},{"why":"GeoCalib estimates camera intrinsics that initialize reconstruction and defines the matched-intrinsics protocol making cross-source pose comparisons fair.","marker":"[44]"},{"why":"The vision-language model that generates all clip-level and segment-level hierarchical annotations; the caption-quality and cross-modal-grounding claims rest on its output.","marker":"[39]"},{"why":"OmniShotCut detects hard cuts and gradual transitions so continuous perspective shots are split into clips of at most 120 seconds without crossing edits.","marker":"[45]"},{"why":"ORB feature matching proposes the temporally distant keyframe pairs that seed panoramic loop-closure candidates.","marker":"[36]"},{"why":"The closed-form SVD rotation that aligns spherical viewing rays provides the geometric verification of whether a candidate is a true co-location.","marker":"[22]"},{"why":"Prior work documenting the scarcity of viewpoint revisits in natural video; it motivates the panoramic revisit-rich subset and frames the paper's contribution.","marker":"[52]"}],"fun_headline_variants":["2,826 hours of real-world footage now pair camera paths with timed captions","Sekai2: 128K clips with camera paths and 982 looping revisits for world models","Long-horizon world modeling gets 2,826 hours of looped, captioned video","World-exploration video now includes reusable revisits for interactive modeling","982 looping paths give world models repeated views for long-horizon consistency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The panoramic revisit claim rests on a single assumption: that two panoramas aligning by rotation with a tiny residual really means the camera returned to the same spot—yet visually repetitive scenes such as malls and corridors, or two spots very close together, can pass the same test without being one location.","fun_headline_variants_meta":{"raw":{"variants":["2,826 hours of real-world footage now pair camera paths with timed captions","Sekai2: 128K clips with camera paths and 982 looping revisits for world models","Long-horizon world modeling gets 2,826 hours of looped, captioned video","World-exploration video now includes reusable revisits for interactive modeling","982 looping paths give world models repeated views for long-horizon consistency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000957,"raw_usage":{"total_tokens":4175,"prompt_tokens":1139,"completion_tokens":3036,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":755,"completion_tokens_details":{"reasoning_tokens":2928}},"tokens_in":755,"tokens_out":3036,"duration_ms":17672,"temperature":1.0,"reasoning_tokens":2928,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:14:58.849123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of accepted loop-closure pairs and fit a full essential matrix (rotation plus translation) to their matched spherical rays instead of a rotation-only alignment. A genuine co-location from the same camera center should show negligible translation; if a substantial fraction of accepted closures show significant parallax—estimated camera-center separation above a few percent of scene depth—the revisit labels are partly spurious and the pose-graph constraints would bias the trajectories. A second ground-truth check: for sequences designed as start-to-end loops, compare the refined start and end camera centers; if endpoint displacement does not drop after loop-closure refinement relative to the original ViPE estimate, the verification is not doing the claimed work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the inherited 47,699 clips and the world-exploration lineage that Sekai2 reprocesses; establishes the predecessor corpus the new release extends."},{"cited_title":"Veicht, P .-E","cited_arxiv_id":null,"evidence_quote":"GeoCalib estimates camera intrinsics that initialize reconstruction and defines the matched-intrinsics protocol making cross-source pose comparisons fair."},{"cited_title":"Rublee, V","cited_arxiv_id":null,"evidence_quote":"ORB feature matching proposes the temporally distant keyframe pairs that seed panoramic loop-closure candidates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closed-form SVD rotation that aligns spherical viewing rays provides the geometric verification of whether a candidate is a true co-location."}],"review_version":2}