{"id":"1379c8dc-d794-4f5a-b950-45c6b5ea35fe","arxiv_id":"2510.20182","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A new benchmark (PEDRA) extracts bird's-eye-view pedestrian trajectories from text- and image-generated videos and finds current video models are plausible at crowd level but let pedestrians merge, collide, or disappear.","lead":"This paper introduces PEDRA, a benchmark that scores text-to-video and image-to-video models on how realistically they simulate crowds of pedestrians. It finds current models produce plausible high-level crowd behavior but often fail at agent-level consistency: people merge, collide, or vanish.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"T2V metric-scale reconstruction is the load-bearing assumption: unvalidated VGGT+DepthPro scale can bias all absolute T2V metrics, and the anthropometric correction only fixes mean height, not depth-dependent errors.","rationale":"The reader's weakest assumption correctly identifies the T2V metric-scale extraction pipeline as the most load-bearing point. The paper's headline claim is two-part: (1) qualitative/behavioral plausibility and (2) quantitative multi-agent statistics. Part (1) is supported by figures, qualitative analysis, and I2V results using known homographies, so it is not directly threatened. Part (2)—especially Table 3's absolute T2V metrics—depends entirely on the unvalidated VGGT+DepthPro+RANSAC scale estimation. The explicit anthropometric correction in Sec 3 is a red flag: it enforces a global mean-height constraint, but if Depth Pro or VGGT has depth-dependent bias, the mean can be correct while individual trajectories, speeds, and inter-agent distances are systematically wrong. The model-dependent discard rates further complicate cross-model comparisons. These issues are addressable by a synthetic ground-truth validation, which is exactly the missing experiment. Since the reader already recommended CONDITIONAL and our concern reinforces that recommendation without overturning the qualitative core, no verdict change is needed; the condition should explicitly require synthetic validation of the T2V reconstruction before the quantitative claims are accepted.","tokens_in":31718,"tokens_out":2930,"duration_ms":28393,"concrete_test":"Render synthetic T2V-style videos with known camera intrinsics/extrinsics, depth, and ground-truth 3D pedestrian trajectories (e.g., Blender crowd simulation with static and slowly moving cameras, various densities). Run the exact VGGT + Depth Pro + RANSAC + FairMOT pipeline used in Sec 3. Compare reconstructed BEV trajectories to ground truth: report median per-agent speed error, per-frame collision-rate error, and flow metric error. Repeat across at least 5 scenes and 2 camera-motion regimes; if median speed error exceeds 10% or collision-rate error exceeds 2 percentage points, Table 3's T2V metrics should be treated as unreliable or recalibrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that video models exhibit an 'effective prior for plausible multi-agent behavior' rests partly on T2V quantitative results (Table 3: velocities, collision rates, flow, NN distance). Those numbers are only meaningful if the reconstructed BEV trajectories are metric-scale accurate. The paper's Sec 3 pipeline estimates a single per-frame scale factor by RANSAC-aligning VGGT normalized depth to Depth Pro metric depth, then applies an anthropometric correction that rescales so mean human height is 1.7 m when the estimated mean falls outside (1.4, 2.0) m. This correction can mask, not remove, systematic bias: a depth error that varies nonlinearly with scene depth or image position leaves the mean height plausible while distorting per-agent positions and speeds, especially for background pedestrians. Moreover, videos are discarded when VGGT and Depth Pro disagree (30% inlier threshold); discard rates differ across models (1.78%–5.0%), so cross-model comparisons in Table 3 are conditioned on different subsets. No synthetic ground-truth validation of the T2V reconstruction is provided, in contrast to I2V where known homographies anchor the comparison. If scale or geometry errors are large or model-dependent, the specific T2V quantitative conclusions are unsupported, even though the qualitative findings and I2V results (which use known homographies) would remain credible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PEDRA, a benchmark protocol for evaluating the realism of multi-agent pedestrian dynamics in text-to-video (T2V) and image-to-video (I2V) generation. For I2V, videos are generated from ETH/UCY start frames and compared with a tracker-consistent ground truth through known homographies. For T2V, a 180-prompt suite spanning density and interaction categories is used, and metric-scale bird's-eye-view trajectories are reconstructed from pixel-space using FairMOT, VGGT camera/depth estimates, Depth Pro metric depth, RANSAC scale alignment, and an anthropometric height correction. Twelve metrics cover kinematics, social interaction, and video fidelity. The central claim is that leading video models have learned an effective prior for plausible multi-agent behavior—translating density and interaction prompts into crowd-level motion—but consistently fail at agent-level integrity, with pedestrians merging, disappearing, or becoming untrackable in dense crowds. The paper also provides a large generated-video dataset and a public evaluation protocol.","tokens_in":1679,"tokens_out":3099,"duration_ms":73920,"significance":"If the measurement pipeline is trustworthy, the paper is a useful step toward evaluating video generation models as implicit pedestrian simulators. Its strengths include a structured T2V prompt suite, a large sampled video/track corpus, and a deliberate I2V design that re-processes ground-truth videos with the same MOT pipeline and uses known homographies. The qualitative findings and the I2V comparisons are credible and align with the claimed trade-off between prompt adherence and agent-level consistency. However, the T2V quantitative results are only as strong as the unvalidated metric-scale reconstruction, and two of the fidelity metrics are computed with the same models used to define the trajectories. These issues affect the quantitative support for the central claim, though they do not invalidate the qualitative conclusions or the I2V results.","major_comments":[{"comment":"The absolute T2V metrics (M_vel, M_acc, M_dist, M_flow, M_nn, M_coll) are computed in the reconstructed metric BEV frame. The reconstruction is validated only by the human-height plausibility check and by discarding samples where VGGT and Depth Pro disagree; there is no synthetic ground-truth validation of the scale-alignment pipeline, in contrast to the I2V task. The anthropometric correction constrains the mean height to 1.7 m, but it cannot correct spatially varying depth bias, which would distort per-agent positions and speeds even when the mean height is plausible. Moreover, the depth-consistency discard rates differ across models (WAN 1.78%, HYV 2.33%, CVX 2.78%, LTX 5.0%, OS 3.22%), so Table 3 compares different retained subsets. This is load-bearing for the T2V quantitative claims. The Limitations paragraph acknowledges label noise, but an acknowledgment is not a validation. Plea","section":"§3, '3D Reconstruction and Scale Estimation'; Table 3; Appendix C"},{"comment":"The video-fidelity metrics are self-referential. M_geo is the mean confidence of VGGT, the same model used to recover camera and depth for the T2V trajectories; M_mot is the FairMOT confidence, and FairMOT is the tracker that determines which tracks enter all metrics. A high M_geo may therefore indicate favorable depth-estimation conditions rather than geometric consistency of the generated video, and M_mot cannot be interpreted independently of the trajectory extraction. Please use an independent reconstruction and tracking model for the fidelity metrics, or at minimum report a correlation analysis and discuss the circularity explicitly.","section":"§3.1, Eq. (20)-(21); Table 1"},{"comment":"All quantitative results are reported as point estimates with no variance, confidence intervals, or significance tests, despite the multi-stage pipeline and the availability of multiple T2V repetitions and multiple I2V start frames. Statements such as 'WAN demonstrates superior geometric consistency' and 'LTX excels on ZARA2' are based on unquantified differences that may be within pipeline noise. Please provide error bars or bootstrap confidence intervals and, where appropriate, test whether model differences are significant.","section":"Tables 2 and 3; §4.2"},{"comment":"The manuscript is internally inconsistent about the I2V sampling procedure. Section 3 says 'we perform multiple inferences until accumulating at least N_gen = 150 unique tracks or 1500 total detections,' while the I2V description and Appendix C state that one video is generated per start frame, with retries only if no trackable agents are produced. If multiple generations per start frame are in fact used, the 'same start distribution' claim is compromised and the resulting EMD comparisons are biased. Please clarify the exact procedure and, if multiple inferences are used, report the number of generations per start frame and any selection criteria.","section":"§3, 'Postprocessing for I2V'; Appendix C"},{"comment":"The collision threshold δ=0.1 m is very small relative to human body width and appears to flag only near-exact overlap of ground-contact points, not actual body collision. Since the collision rate is a headline T2V result, please justify the threshold or report sensitivity to δ over a physically motivated range (e.g., 0.1–0.5 m).","section":"Appendix A.2, Collision Rate"}],"minor_comments":[{"comment":"Notation is inconsistent: 'M_DTW_int-div' appears in Table 1 and Eq. (9) in different forms. Please unify.","section":"Table 1"},{"comment":"The expression for Internal Diversity is garbled: the binomial coefficient should be binom{N}{2}. Please fix the math.","section":"Appendix A.1, Eq. (9)"},{"comment":"Typos: 'simulaitons' in the metric introduction and 'compue' in the Velocity paragraph.","section":"Appendix A"},{"comment":"'Appenix Fig. 10' in the main text is a typo for 'Appendix Fig. 10'.","section":"Figure 10 caption"},{"comment":"The note that bold values do not indicate desirability is confusing. Use a different visual emphasis (e.g., color or arrows) so that interpretation is not tied to boldness.","section":"Table 3"},{"comment":"References [96] and [97] appear to be the same VBench-2.0 paper; consider merging them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong benchmark contribution in scope, and the qualitative/I2V parts are solid. My recommendation is driven by the unvalidated T2V metric-scale reconstruction and the self-referential fidelity metrics. Both are fixable within the manuscript's scope, so I would not reject. Please ask the authors to add a synthetic validation or sensitivity analysis, clarify the I2V sampling procedure, and report uncertainty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read PEDRA. It is the first benchmark I know that evaluates multi-agent pedestrian dynamics in generated video rather than single-subject realism, and that gap is real. The qualitative findings—models translate \"crowded/directional/converging\" prompts into crowd-level behavior but merge or drop individual agents—are convincing and well documented. The I2V arm, using known ETH/UCY homographies, is methodologically sound; comparing generated trajectories to the same MOT pipeline run on ground-truth video is the right call. This part of the paper deserves to be read.\n\nThe T2V arm is where I get cautious. The whole quantitative superstructure rests on VGGT + Depth Pro + RANSAC scale alignment, capped by the 1.7 m height correction. The stress-test note lands: that correction fixes mean height, not depth-dependent errors, so per-agent positions, speeds, collision rates, and flow values can be biased in ways that are hard to detect. The paper acknowledges \"label noise\" in Limitations but gives no synthetic ground-truth validation where camera and scene geometry are known. The discard rates also differ across models (1.78%–5.0%), so cross-model T2V comparisons are conditioned on different subsets. Because the headline claim about \"effective priors\" rests partly on T2V numbers, I would not treat those absolute values as reliable until the pipeline is validated. The I2V results and the qualitative conclusions stand on their own.\n\nTwo smaller issues: tables report point estimates with no variance or confidence intervals—900 videos per model makes that an easy fix—and the fidelity metrics are self-referential: VGGT confidence scores VGGT-reconstructed scenes, and FairMOT confidence scores FairMOT tracks. That weakens those columns, though they are clearly labeled. Code and data are promised but not yet released; for a benchmark paper that is a condition for fully trusting the numbers.\n\nNet: a serious, useful contribution with a load-bearing but addressable weakness. I would send it to peer review with the expectation of major revision: validate T2V reconstruction against synthetic ground truth, add confidence intervals, release code and prompts. If the authors do that, this becomes a standard reference. It deserves a serious referee; it is not ready as-is. I would bring it to reading group and cite it once the code lands.","headline":"A genuinely useful first benchmark for multi-agent pedestrian dynamics in video generation, with a solid I2V arm and an interesting but unvalidated T2V reconstruction pipeline that should not be trusted for absolute numbers yet.","tokens_in":32556,"tokens_out":2194,"would_cite":true,"duration_ms":20422,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Leading video generation models have internalized crowd-level pedestrian behavior — density, flow, collision avoidance — but cannot keep individual pedestrians distinct, frequently merging or erasing them.","keywords":["pedestrian dynamics","video generation models","world simulators","multi-agent behavior","crowd simulation","bird's-eye view trajectories","benchmarking","trajectory extraction"],"falsifier":"Feed the pipeline a synthetic video rendered with known camera intrinsics, known world-scale human heights, and known pedestrian trajectories. If the recovered bird's-eye trajectories carry mean speed error above a few percent, or if the anthropometric correction fires when the true heights are already correct, then the metric-scale extraction is biased and the quantitative claims about speed and social spacing are not settled.","tokens_in":31568,"feed_emoji":"🚶","tokens_out":7638,"duration_ms":63451,"temperature":0.7,"pith_summary":"This paper asks whether video-generation models can act as implicit pedestrian simulators, producing not just believable images but believable crowd motion. Testing five text-to-video and image-to-video models, the authors find that the models carry a genuine crowd-level prior: they translate prompts about sparse or crowded scenes and directional or converging movement into measurable changes in population, speed, and collision rate, and they reproduce the basic crowd-dynamics relation in which people slow down as density rises. At the level of individual agents, however, the prior breaks down: pedestrians merge into one another instead of avoiding collisions, vanish mid-trajectory, and degrade into untrackable pixelated masses in dense crowds. To make this assessment, the paper contributes a method for reconstructing metric-scale bird's-eye-view trajectories from synthetic video with no known camera parameters, using structure-from-motion, metric depth estimation, and scale alignment. The stakes are practical: if these priors hold, video models could populate scenes and generate statistically plausible crowd behavior without hand-tuned simulation rules, while the agent-integrity failures mark exactly where such models cannot yet be trusted.","feed_headline":"Crowd videos from AI look real, but pedestrians merge and vanish","feed_subtitle":"Benchmark of five text-to-video models finds density and flow are right, but individual pedestrians are unstable.","key_machinery":"The load-bearing piece is a trajectory-extraction pipeline that turns pixel-space detections into metric bird's-eye-view trajectories in synthetic scenes with unknown cameras: an off-the-shelf multi-object tracker finds pedestrians, structure-from-motion estimates per-frame camera pose and scene geometry, a metric-depth estimator supplies real-world scale, a RANSAC alignment fits per-frame scale factors by minimizing a Huber loss between the two depth maps, and an anthropometric check re-scales the mean estimated human height to 1.7 meters whenever it falls outside the 1.4–2.0 meter range. Around this pipeline sits a twelve-metric evaluation protocol spanning trajectory kinematics (velocity,","core_discovery":"The paper's central claim is that leading video diffusion models have learned an effective prior for plausible multi-agent pedestrian behavior even though no pedestrian model is built into them. Conditioned on text, they map density and interaction cues onto coherent crowd-level motion: crowded prompts produce larger populations and higher collision rates, directional prompts produce faster walking, and the flow-density curve follows the expected decreasing trend. Conditioned on a start frame, they reproduce the approximate spatial distribution and nearest-neighbor spacing of a ground-truth scene. The same models, however, break at the agent level: pedestrians merge or spontaneously disappea","pith_inferences":["A simple testable extension would make agent integrity a first-class metric: count track terminations that occur while a pedestrian is still in frame, separate from collisions. The paper documents such disappearances qualitatively but does not give them a dedicated score.","The 1.7 m height re-scaling means the absolute speed numbers inherit the depth estimator's prior. Validating the pipeline on synthetic scenes with known camera intrinsics and known human heights would harden or soften every quantitative comparison in the paper.","The paper's own Limitations section concedes that the multi-stage extraction pipeline can inject label noise, particularly in metric-scale estimation; the anthropometric correction is a mitigation, not a proof of accuracy.","The five-second horizon and the observed time-lapse artifacts suggest that longer generations would compound the merging and vanishing failures; a 10–30 second extension would test whether the crowd-level prior degrades with duration."],"forward_implications":["Text-to-video generation can act as a partial pedestrian simulator: it populates scenes and produces crowd-level motion from a natural-language description, without hand-written rules or parameter tuning.","The prompt-to-behavior link is measurable: density and interaction words shift population, speed, collision, and flow statistics in consistent directions across five different models.","Agent permanence is the key bottleneck: any application that counts people or follows individuals across time — evacuation studies, human-robot interaction, exact trajectory prediction — will fail even when the crowd looks right.","Model design and training data create a real trade-off between single-subject clarity and crowd fidelity; models that filter crowded footage to improve motion clarity show the worst dense-scene behavior.","The evaluation protocol, and the trajectory-reconstruction method behind it, gives video-generation developers a concrete target: improve agent-level consistency without losing the crowd-level prior."],"fun_headline_variants":["AI crowds look real until pedestrians merge and vanish","Video AI: crowd flow realistic, but walkers blink out","AI video crowds pass realism test, fail individual stability","Crowd videos: density right, pedestrians unreliable"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That the automatic pipeline converting generated pixels into metric bird's-eye-view trajectories — structure-from-motion, metric depth, scale alignment, and a height-based correction — produces world-scale positions accurate enough that the measured speeds, densities, collision rates, and spacing reflect the model's behavior rather than measurement error.","fun_headline_variants_meta":{"raw":{"variants":["AI crowds look real until pedestrians merge and vanish","Video AI: crowd flow realistic, but walkers blink out","AI video crowds pass realism test, fail individual stability","Crowd videos: density right, pedestrians unreliable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1084,"prompt_tokens":689,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":331}},"tokens_in":433,"tokens_out":395,"duration_ms":4513,"temperature":1.0,"reasoning_tokens":331,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:28:33.287784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the pipeline a synthetic video rendered with known camera intrinsics, known world-scale human heights, and known pedestrian trajectories. If the recovered bird's-eye trajectories carry mean speed error above a few percent, or if the anthropometric correction fires when the true heights are already correct, then the metric-scale extraction is biased and the quantitative claims about speed and social spacing are not settled.","supporting_citations":[],"review_version":1}