{"id":"539ceacf-f153-4eb4-885b-01c6be2bc419","arxiv_id":"2411.17376","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A detection-input transformer with synthetic pretraining and weakly-supervised fine-tuning achieves state-of-the-art or competitive pedestrian forecasting while avoiding person ID annotations.","lead":"This paper presents RealTraj, a trajectory forecasting framework that takes raw detections instead of tracked trajectories, pretrains on synthetic data with self-supervised tasks, and fine-tunes on real data using only detection boxes, cutting the need for person ID annotations and improving robustness to perception errors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Weakly-supervised nearest-detection loss can track the wrong pedestrian in dense crowds; the paper does not bound this failure mode.","rationale":"The reader's weakest assumption aligns with the most load-bearing risk in the paper. RealTraj's headline contribution is that it can fine-tune trajectory forecasting using only ground-truth detections, dropping person ID annotations. Eq. (4) implements this as a per-timestep nearest-detection loss. The technical weakness is that the nearest detection at each timestep is chosen independently, so the supervision target is not guaranteed to be the same pedestrian across timesteps. In crowded or intersecting scenarios, the target can switch, making the loss a biased estimator of the true future trajectory. The acceleration regularizer mitigates oscillation but does not enforce identity correspondence; it can even encourage a smoothly interpolated path through different pedestrians. The paper's experiments show competitive results on standard benchmarks, but these do not rule out the failure mode because the benchmarks are relatively sparse, and no analysis is provided on the association correctness of the nearest-detection loss. Other concerns, such as the asymmetric corruption protocol and missing error bars, are secondary and do not directly challenge the central claim. A concrete test can settle the issue by measuring the identity-match rate and comparing against an oracle-supervised variant on dense scenes. If the oracle variant is significantly better, the weak supervision is not providing the intended signal, and the claim of reducing annotation cost is weakened. Therefore, the conditional verdict is appropriate; the paper should either provide such an analysis or constrain its claims to sparse scenes.","tokens_in":18132,"tokens_out":9040,"duration_ms":86894,"concrete_test":"During weakly-supervised fine-tuning on JRDB, for each target pedestrian and each future timestep, record whether the nearest ground-truth detection d^c_t belongs to the same pedestrian as the target (using the ground-truth IDs that are available in the dataset but not used for supervision). Then retrain the model with an oracle loss that uses the true target position instead of d^c_t, and compare ADE/FDE on a dense subset (e.g., scenes with at least 10 pedestrians within a 5m radius of the target). If the oracle variant improves ADE by more than 5% relative to the weakly-supervised variant, the nearest-detection assumption is substantially violated on dense data, and the central claim that person IDs can be eliminated needs to be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's central novelty is training without person IDs via the weakly-supervised loss in Eqs. (3)-(4). This loss uses, at each future timestep, the closest ground-truth detection d^c_t to the predicted point as the supervision target. Because d^c_t is selected independently per timestep without any identity consistency constraint, the target can switch from the true pedestrian to a nearby one when paths cross or in dense crowds. The acceleration regularizer (Eq. 5) only penalizes large second-order differences; it does not penalize a smooth drift that follows different pedestrians, since a trajectory that continuously tracks the local density can have low acceleration variance. Consequently, the loss is not an unbiased estimator of the target's true future trajectory; it can converge to a solution that chases whichever pedestrian is locally closest. The paper does not report any diagnostic measuring how frequently the nearest detection is the correct identity, nor does it evaluate performance as a function of crowd density or trajectory crossing. If this failure mode is prevalent on dense real data, the claimed ability to eliminate person ID annotations would not transfer, and the reported gains on standard benchmarks could be an artifact of relatively sparse scenes. This is the most load-bearing concern because it directly targets the method's key contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RealTraj is a pedestrian trajectory forecasting framework that takes as input a set of detections without person identities. It consists of Det2TrajFormer, a transformer encoder that processes detection tokens, and two training phases: self-supervised pretraining on synthetic ORCA trajectories with unmasking, denoising, and person-ID reconstruction pretext tasks, followed by weakly-supervised fine-tuning on real ground-truth detections using a nearest-detection regression loss with an acceleration regularizer. The paper reports experiments on JRDB, JTA, ETH-UCY, SDD, and TrajImpute, evaluating robustness to miss-detections, localization errors, and identity switches, few-shot performance, and full-supervision comparisons. The central claims are that the framework reduces ID annotation costs, improves robustness to perception errors, and achieves state-of-the-art or comparable forecasting accuracy on several datasets.","tokens_in":59,"tokens_out":8855,"duration_ms":178227,"significance":"If validated, the framework addresses an important practical gap: most forecasting models require clean tracked trajectories with consistent IDs, while real perception pipelines produce noisy detections. The paper's strengths include proposing a unified solution to three limitations at once, synthetic pretraining with multiple pretext tasks, and a broad set of experiments covering robustness, few-shot regimes, and ablations. The identity-switch invariance follows naturally from the detection-based input representation, which is a clean architectural design choice. However, the central weakly-supervised objective—the closest-detection target—has a potential failure mode in dense scenes, and the reported experiments do not yet characterize it. The paper would be strengthened substantially by a diagnostic that measures how often the nearest detection is the true target pedestrian.","major_comments":[{"comment":"The weakly-supervised loss selects the nearest ground-truth detection at each future timestep as the regression target. Because the selection is made independently per timestep without any identity constraint, the target can switch from the target pedestrian to a nearby pedestrian when trajectories cross or in dense crowds. The acceleration regularizer in Eq. (5) only penalizes large second-order differences and cannot prevent a smooth drift that follows different pedestrians. This is load-bearing because it is the mechanism that eliminates person ID annotations. The paper provides no diagnostic for how often d^c_t is the true target pedestrian, nor a density-stratified analysis. Please add such a diagnostic (e.g., the identity-match rate of the nearest detection on the datasets used), and an ablation comparing Eq. (4) with an oracle identity-based target. If the identity-match rate is low in dense scenes, the claims about reducing ID annotation costs need to be revised.","section":"3.4, Eqs. (3)–(5)"},{"comment":"The robustness comparison is not controlled. For miss-detections, the proposed model receives zero-filled detections while the baselines receive linearly interpolated detections; for identity switches, the baselines receive swapped IDs while RealTraj is unaffected by construction because its input has no IDs. As a result, the comparison in Fig. 4 does not isolate model robustness from input preprocessing or architectural assumptions. To support the claim of robustness, the authors should either run all methods on identical corrupted inputs (e.g., the same interpolated or detector-output detections), or explicitly justify why different input treatments are appropriate and add a comparison on realistic detector/tracker outputs.","section":"4.4, Fig. 4"},{"comment":"All metrics are reported as single runs without variance. The few-shot experiment in Table 2 randomly selects subsets, so results will vary with the sample; several reported differences are small (e.g., 0.43 vs 0.45 on JRDB at 0.1%). Without standard deviations or repeated seeds, the claimed improvements are not statistically grounded. Please report means and standard deviations over at least three seeds for the few-shot experiments, and ideally for the main comparisons in Table 3.","section":"Tables 2 and 3"},{"comment":"The abstract states that 'the method outperforms state-of-the-art trajectory forecasting methods on multiple datasets.' This holds for the fully-supervised variant on JRDB, JTA, and SDD, but the weakly-supervised variant—the paper's main contribution—does not outperform on ETH-UCY (e.g., minADE20=0.26 vs EqMotion's 0.21; minFDE20=0.43 vs 0.35). The claim should be qualified to distinguish the fully-supervised and weakly-supervised variants, or to state the specific datasets for which the weakly-supervised variant is superior.","section":"Abstract and Table 3"}],"minor_comments":[{"comment":"The schedule of the loss weights states that (α, β, γ) changes from (1,0,0) to (0,100,0.1). The value β=100 is surprisingly large and may be a typo; please clarify and, if it is intentional, justify the magnitude.","section":"Supplementary, Sec. 9"},{"comment":"The w/ Syn. indicators (✓/✗) are embedded within the numeric rows, making the table difficult to parse; please place them in a separate column as described in the caption.","section":"Table 2"},{"comment":"The text says '2K synthetic trajectories' while Fig. 5(b) shows ablation up to 5000 sequences; please clarify the total number of sequences used in the default setting.","section":"Sec. 4.3 and Fig. 5(b)"},{"comment":"The first row (the no-pretraining baseline) is listed with no main task and no pretext tasks; please specify how this model is trained (e.g., from scratch with only the fine-tuning loss) so that the comparison is interpretable.","section":"Table 4"},{"comment":"The limitation section mentions only that pedestrians must be detected in the last observed frame; it does not discuss the identity-association ambiguity of the weakly-supervised target in dense crowds, which is a more direct limitation of the proposed fine-tuning scheme.","section":"Sec. 6"},{"comment":"The notation X ∈ R^{K T_obs × 2} is inconsistent with X_t ∈ R^{K×2}; consider using a product space or clarifying the reshaping.","section":"Sec. 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution, but the central claim of removing person ID annotations hinges on the nearest-detection loss being a faithful surrogate for the true future trajectory. The missing diagnostic is the main risk; if the identity-match rate is low in dense scenes, the method's advantage may be limited to sparse settings. I would encourage the editor to require the diagnostic as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you work on trajectory forecasting or perception-robust prediction. It is a practical, well-executed combination: a transformer that consumes raw detection sets (not tracked trajectories), pretrained on synthetic data with three pretext tasks, then fine-tuned with a weak nearest-detection loss and an acceleration regularizer. The main thing to check before trusting the numbers is the behavior of that weakly supervised loss in dense scenes; the paper does not stress-test it.\n\nWhat it does well: Det2TrajFormer is inherently invariant to identity switches because it never sees person IDs across frames. The three pretext tasks (unmasking, denoising, ID reconstruction) are sensible, and the ablations show each contributes. The experimental coverage is broad: JRDB, JTA, ETH-UCY, SDD, and the TrajImpute missing-data benchmark. The few-shot results are the most interesting part; at 0.1–1% labels, RealTraj beats supervised baselines that get full trajectories and IDs, which is a genuine practical claim. The paper is honest about its own limitation (only pedestrians visible in the last frame) and does not oversell its novelty.\n\nSoft spots, in proportion:\n\n1. The closest-detection loss (Eqs. 3–4) can attach predictions to the wrong pedestrian when paths cross. The acceleration regularizer suppresses oscillation but not systematic drift toward a nearby person. The paper never reports how often the matched detection is the correct target, nor does it split performance by crowd density or crossing rate. This is load-bearing because the headline claim is fine-tuning without IDs. It does not sink the paper—the empirical wins on standard benchmarks suggest the loss mostly works—but a referee should ask for a diagnostic or a synthetic experiment that measures target-switching.\n\n2. The robustness comparison is asymmetric: for miss-detections, baselines get linearly interpolated inputs while RealTraj gets zero-filled coordinates. Zero-filling is arguably harsher, so this is not a bias in RealTraj’s favor, but the protocol is not matched and should be clarified.\n\n3. All reported metrics are single-run, no variance. This is common but matters when claiming SOTA.\n\n4. Code is promised but not yet available, so results cannot be independently reproduced.\n\nThe related work citation pattern looks fine, and the author self-citations are relevant prior work on robustness.\n\nWho is this for? Researchers and engineers applying trajectory forecasting in real systems (robotics, autonomous driving, surveillance), and anyone working on SSL or weak supervision for motion prediction. It is a systems-oriented empirical paper, not a new theoretical paradigm.\n\nRecommendation: send to a serious referee. The paper should be accepted after revision if the authors add an analysis of target-switching in the weak loss and run the main tables with multiple seeds.","headline":"RealTraj is a solid empirical paper showing a detection-only transformer with synthetic pretraining and weakly supervised fine-tuning can hold up against detection and tracking noise; the main open question is whether the closest-detection loss silently tracks the wrong pedestrian in dense crowds.","tokens_in":18855,"tokens_out":3062,"would_cite":true,"duration_ms":35004,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RealTraj claims pedestrian trajectory forecasting can be trained and run on raw detections alone, without person IDs, and still match or beat fully supervised state-of-the-art models.","keywords":["pedestrian trajectory forecasting","detection-only input","self-supervised pretraining","weakly supervised fine-tuning","transformer encoder","perception error robustness","synthetic data pretraining","person identity annotation cost"],"falsifier":"Run the weakly supervised variant on a dense sequence where pedestrians frequently cross paths, and measure how often the predicted trajectories switch to following the wrong person's detections after a crossing, compared with the fully supervised variant; a sizable switch rate would show the closest-detection assumption breaks in crowded scenes.","tokens_in":17908,"feed_emoji":"🚶","tokens_out":8361,"duration_ms":72468,"temperature":0.7,"pith_summary":"RealTraj asks whether pedestrian trajectory forecasting can be made practical for real-world deployment by removing two costly assumptions: that perfect tracked trajectories are available, and that every pedestrian must carry a persistent identity label. The paper proposes a two-phase scheme: self-supervised pretraining on synthetic simulator data with three auxiliary tasks (unmasking, denoising, person-identity reconstruction), followed by weakly supervised fine-tuning on real data using only ground-truth detections, where the nearest future detection to each prediction serves as the target and an acceleration regularizer suppresses jitter. The paper reports that this detection-only, ID-free training matches or beats fully supervised state-of-the-art models on JRDB, JTA, and SDD, stays competitive on ETH-UCY, and degrades far less under simulated miss-detections, localization noise, and identity switches. If the claims hold, forecasting models can be trained without expensive person-ID annotations and can run directly on detector output instead of clean tracks.","feed_headline":"Detection-only training beats fully supervised trajectory forecasters","feed_subtitle":"Synthetic pretraining plus detection-only fine-tuning reaches state-of-the-art accuracy while ignoring tracking errors.","key_machinery":"The load-bearing design is Det2TrajFormer, a Transformer encoder whose input is a per-frame set of detections (positions only, with no IDs) plus learnable query tokens that read out future positions. Its robustness comes from the removal of identity information from the input stream, making identity-switch errors nonexistent by construction, and from three pretext heads used during synthetic pretraining: an unmasking head that reconstructs masked detections, a denoising head that removes added Gaussian noise, and a person-ID reconstruction head that forces the encoder to associate detections of the same pedestrian across frames. The coupling that makes weak supervision work is the closest-detection loss of Eq. (3)–(4), with the acceleration regularizer of Eq. (5) smoothing the resulting trajectories.","core_discovery":"The central claim is that pedestrian trajectory forecasting can be driven entirely by detections, with no person identities ever provided or inferred during training. The proposed model, Det2TrajFormer, is a Transformer encoder that takes an unordered set of past box positions per frame, predicts the future positions of any pedestrian designated as target by a translation of the input frame, and is pretrained on synthetic trajectories with three auxiliary objectives: reconstructing masked detections, denoising corrupted ones, and reconstructing person-identity embeddings so the encoder learns to associate detections across frames. Fine-tuning then uses a weakly supervised loss that matches each predicted future point to the closest ground-truth detection at that timestamp and adds an acceleration-regularization term to prevent oscillation. On the reported benchmarks the fully supervised variant achieves the best ADE/FDE on JRDB, JTA, and SDD, the weakly supervised variant matches fully supervised performance on JRDB and SDD, and under synthetic corruption of 20–80% of inputs the method's error grows far more slowly than that of the leading baselines.","pith_inferences":["The paper leaves open whether the same detection-only principle extends to pedestrians not visible in the last frame; a testable variant would add a no-detection branch to the weak-supervision loss so the model could forecast for occluded agents.","The identity-free formulation could be embedded inside a multi-object tracker as a forward motion prior, where its robustness to identity switches might improve association across occlusion gaps, though the paper does not test this.","Because the pretext tasks each target one corruption type, one could probe whether matching the corruption ratio used in pretraining to the corruption level expected at deployment changes the robustness curve; the paper ablates the ratio but not this alignment."],"forward_implications":["Forecasting models can be deployed directly on detector output, eliminating the tracking stage as a prerequisite and the error propagation that comes with it.","Person-ID annotation becomes unnecessary for fine-tuning, cutting a major data-preparation cost in building trajectory datasets.","Synthetic pretraining with corruption-augmented pretext tasks transfers to real scenes, so real-data collection can be kept small.","The acceleration-regularized closest-detection loss is a viable weak supervision signal, giving performance on par with fully supervised training on several benchmarks."],"supporting_citations":[{"why":"Supplies the synthetic crowd simulator used to generate all pretraining trajectories; the synthetic-data half of the two-phase scheme rests on it.","marker":"[70]"},{"why":"Supplies the strongest fully supervised baseline; its performance under clean and corrupted inputs sets the bar RealTraj must beat.","marker":"[79]"},{"why":"Supplies the other leading baseline, adapted to trajectory-only inputs to ensure fair comparison in every experiment.","marker":"[61]"},{"why":"Supplies the missing-data benchmark used to evaluate robustness to miss-detections; the hard-split results support the robustness claim.","marker":"[13]"},{"why":"Supplies the self-supervised pretraining formulation and the winner-take-all multi-future training strategy RealTraj adopts for its forecasting head.","marker":"[11]"}],"fun_headline_variants":["No IDs, no problem: detection-only trajectory forecasting","Detection-only training beats fully supervised for path prediction","Synthetic pretraining plus detection fine-tuning tops full supervision","Robust to tracking noise: forecasts from raw detections"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fine-tuning loss assumes the closest future detection to each predicted point belongs to the same pedestrian; in crowded scenes with crossing paths this can lock predictions onto the wrong person's detections.","fun_headline_variants_meta":{"raw":{"variants":["No IDs, no problem: detection-only trajectory forecasting","Detection-only training beats fully supervised for path prediction","Synthetic pretraining plus detection fine-tuning tops full supervision","Robust to tracking noise: forecasts from raw detections"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1937,"prompt_tokens":961,"completion_tokens":976,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":924}},"tokens_in":577,"tokens_out":976,"duration_ms":9468,"temperature":1.0,"reasoning_tokens":924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:10:41.840155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the weakly supervised variant on a dense sequence where pedestrians frequently cross paths, and measure how often the predicted trajectories switch to following the wrong person's detections after a crossing, compared with the fully supervised variant; a sizable switch rate would show the closest-detection assumption breaks in crowded scenes.","supporting_citations":[{"cited_title":"van den Berg, Stephen J","cited_arxiv_id":null,"evidence_quote":"Supplies the synthetic crowd simulator used to generate all pretraining trajectories; the synthetic-data half of the two-phase scheme rests on it."},{"cited_title":"Eq- Motion: Equivariant Multi-agent Motion Prediction with In- variant Interaction Reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies the strongest fully supervised baseline; its performance under clean and corrupted inputs sets the bar RealTraj must beat."},{"cited_title":"Social-Transmotion: Promptable Human Trajectory Prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the other leading baseline, adapted to trajectory-only inputs to ensure fair comparison in every experiment."}],"review_version":1}